Apparatus for providing calibration data, camera system and method for obtaining calibration data

ABSTRACT

An apparatus comprises a first interface for receiving a plurality of partially overlapping images of an object from a corresponding plurality of cameras being arranged along a first and a second direction according to a camera pattern. The apparatus comprises an analyzing unit configured for selecting at least one corresponding reference point in an overlap area of a set of images, and for determining displacement information along the first and the second direction of the reference point in each of the other images of the set of images. A misalignment of the plurality of images along the first and the second direction is compensated by the displacement information so as to obtain aligned images. The apparatus comprises a determining unit configured for determining offset information between principal points of the plurality of cameras using at least three aligned images. The apparatus comprises a second interface for providing calibration data based on the displacement information and based on the offset information. The calibration data allows for calibrating the plurality of images so as to comply to the camera pattern.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of copending InternationalApplication No. PCT/EP2018/025077, filed Mar. 28, 2018, which isincorporated herein by reference in its entirety, and additionallyclaims priority from European Application No. EP 17163601.2, filed Mar.29, 2017, which is also incorporated herein by reference in itsentirety.

The present invention is directed to an apparatus for providingcalibration data, in particular, for calibrating images from a pluralityof cameras. The present invention is further directed to a camerasystem, to a method for obtaining calibration data and to anon-transitory storage medium having stored thereon a computer program.The present invention is further directed to a multi-camerarectification for planar and non-planar camera arrays.

BACKGROUND OF THE INVENTION

Multi-camera systems are the base for modern visual effects like refocusor novel-view-synthesis as well as geometric analysis and depthreconstruction. An important requirement for high quality and highefficiency image processing is precise camera positioning andorientation. Due to mechanical deviations, like small deviations from anideal orientation, this requirement is, in general, not met.

The general task of camera calibration is to estimate values for theinternal (intrinsic) and external (extrinsic) parameters of a (digital)camera. The most important intrinsic parameters are focal length andprincipal point of the camera. Extrinsic parameters denote position andorientation of a camera. In the context of media production, cameracalibration is, for example, needed when one likes to add computergenerated (CG) objects to the scene: When the physical camera is movingthrough the scene, the computer generated objects shall moveconsistently with the real world objects. Therefore, the path of thecamera through the scene needs to be known precisely to control the pathof a CG camera in a 3D environment. This process is also known as cameratracking or match-moving.

In case of stereoscopic productions, camera images need to be alignedsuch that corresponding image points are in the same image line. Such aprecise alignment is, in general, not possible by mechanical setups andneeds to be done by digital image processing. This process is also knownas rectification. A typical stereoscopic production usually involves twoidentical cameras precisely mounted on a rig. This initial, mechanicalcalibration ensures that camera images are already close to therectified state. A more general approach for stereoscopic production ispresented in [1]. In this work, three cameras on a line are used tovirtually manipulate the interocular distance of the stereo system.Therefore, the stereo calibration process is extended towards a threecamera system.

A widely used camera model is the pinhole camera. It is typicallydescribed by up to 11 individual parameters composed in the projectionmatrix P. These 11 parameters are focal length (horizontal andvertical), skew, principal point (horizontal and vertical), orientationcomprising 3 angular components and location comprising three componentsof a 3D coordinate. This matrix may also be represented as a product ofthe camera matrix K, the rotation matrix R and the position C of thecamera given as a 3D world coordinate. More parameters such as to modellens-distortion can be added if needed yielding more than 11 parameters.

Estimating intrinsic and extrinsic parameters for a single camera or asystem of cameras can be done using calibration charts or based purelyon images. In the first case, usage of calibration charts allows thereconstruction of camera parameters in metric units; for example, afocal length of the camera may be given in a mm unit, i.e., millimeters.Such a calibration needs to be done right before or after capturing ashot on a set. The calibration chart is placed in front of the camerasand each camera captures an image of the chart. It is important that thephysical properties such as focal length, orientation or position of amulti-camera system do not change between the calibration shot and theshot one wants to capture and process. Otherwise, calibration might beunreliable. However, many algorithms known for calibrating multi-camerasystems entail that the calibration chart is completely visible in everyimage. For large camera arrays, this is challenging.

The second case is widely known as self-calibration or auto-calibrationand typically needs at least two images from a set of stereo cameras ortwo images from the same camera taken at different positions. Theadvantage of self-calibration compared to methods using calibrationcharts is that the cameras can be calibrated even if the capturedcontent was not specifically intended for calibration. This is, forexample, the case when the content was not intentionally shot forintegration of CG elements but integrating CG elements is needed inpost-production. In contrast to chart-based methods, camera parameterscan also be re-estimated if physical parameters change within a runningcapture. Self-calibration methods estimate camera parameters innon-metric units. As self-calibration algorithms exploit thecorrespondence between several images, such algorithms typically involvethat at least two or more images have overlapping image areas.Therefore, parts of the captured scene need to be visible in more thanone image.

SUMMARY

According to an embodiment, an apparatus may have: a first interface forreceiving a plurality of partially overlapping images of an object froma corresponding plurality of camera positions being arranged along afirst and a second direction according to a two-dimensional orthree-dimensional camera pattern, wherein patterns having camerapositions that differ in two directions are two-dimensional patterns andpatterns having camera positions that differ in three directions arethree-dimensional patterns; an analyzing unit configured for selectingat least one corresponding reference point in an overlap area of a setof overlapping images, and for determining a displacement informationalong the first and the second direction of the reference point in eachof the other images of the set of images, wherein a misalignment of theplurality of images along the first and the second direction iscompensated by the displacement information so as to obtain alignedimages; a determining unit configured for determining an offsetinformation between principal points at the plurality of camerapositions using at least three aligned images; and a second interfacefor providing calibration data based on the displacement information andbased on the offset information, the calibration data allowing forcalibrating the plurality of images so as to comply to the camerapattern; wherein the analyzing unit is configured to determine thedisplacement information minimizing an error of a first minimizationcriteria, wherein for a 2D camera pattern the first minimizationcriteria is based on the determination rule;

$\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{{{v\left( {{H\left( {x_{p},{C_{p,}C_{q}}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},{C_{p,}C_{q}}} \right)} \cdot m_{p,q}^{(q)}} \right)}}}}$with  x = ∑_(l)x_(l)²

wherein the determining unit is configured to determine the offsetinformation minimizing an error of a second minimization criteria,wherein for the 2D camera pattern the second minimization criteria isbased on the determination rule

$\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{{{\left( {{u\left( {{H\left( {x_{p},{C_{p,}C_{q}}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},{C_{p,}C_{q}}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\left( {{u\left( {{H\left( {x_{p},{C_{p,}C_{r}}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{r},{C_{p,}C_{r}}} \right)} \cdot m_{p,q,r}^{(r)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}}}}$     with  x = ∑_(l)x_(l)²

wherein argmin denotes the minimization criteria and p, q and r denoteindices of individual cameras. C_(p), C_(q), C_(r) denote camerapositions of cameras p, q and r, d(C_(p)−C_(q)) is a distance betweenthe camera positions, x_(p), x_(q) and x_(r) are vectors with parametersto be determined so as to minimizing the error for camera p, q and r,H(⋅) is a function providing a homography matrix undistorting pointsaccording to a parameter vector x_(p) and camera positions C_(p) andC_(q), m_(p,q)=[m^((p)) _(p,q) m^((q)) _(p,q)] denotes a set ofreference points wherein m^((p)) _(p,q) corresponds to the respectivereference points in camera p, m_(p,q,r)=[m^((p)) _(p,q,r) m^((q))_(p,q,r) m^((r)) _(p,q,r)] denotes a set of reference points whereinm^((p)) _(p,q,r) corresponds to the respective reference points incamera p, Functions u(⋅) and v(⋅) extract the horizontal/verticalcomponent of an image point respectively, the set Ω includes all pairsof partially overlapping images; the set ψ includes triplets ofpartially overlapping images; such that each of the N cameras consideredin the minimization problem is comprised in at least one pair of camerascontained in Ω and at least in one triplet of cameras contained in ψ;and/or wherein the analyzing unit is configured to determine thedisplacement information minimizing an error of a linearized firstminimization criteria, wherein for a 2D camera pattern the linearizedfirst minimization criteria is based on the determination rule;

$\mspace{79mu} {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}}}} + b_{p,q}}\mspace{14mu}$$\mspace{79mu} {{{wherein}\mspace{14mu} \left( {{v\left( {{H\left( {x_{p},{C_{p,}C_{q}}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},{C_{p,}C_{q}}} \right)} \cdot m_{p,q}^{(q)}} \right)}} \right)}^{2} \approx {\quad{{\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}} + b_{p,q}}\;}}$

wherein the determining unit is configured to determine the offsetinformation minimizing an error of a linearized second minimizationcriteria, wherein for the 2D camera pattern the linearized secondminimization criteria is based on the determination rule

$\mspace{79mu} {{\underset{x_{1},{x_{2}x_{3}},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{\begin{bmatrix}J_{p,q,r}^{(p)} & \begin{matrix}J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{matrix}\end{bmatrix} \cdot \begin{bmatrix}x_{p}^{T} \\\begin{matrix}x_{q}^{T} \\x_{r}^{T}\end{matrix}\end{bmatrix}}}} + b_{p,q,r}}$$\mspace{79mu} {{{wherein}\left( {{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(r)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}} \right)}^{2} \approx {\quad{{\begin{bmatrix}J_{p,q,r}^{(p)} & \begin{matrix}J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{matrix}\end{bmatrix} \cdot \begin{bmatrix}x_{p}^{T} \\\begin{matrix}x_{q}^{T} \\x_{r}^{T}\end{matrix}\end{bmatrix}} + b_{p,q,r}}}}$

wherein argmin denotes the minimization criteria p, q and r denoteindices of individual cameras, C_(p), C_(q), C_(r) denote camerapositions of cameras p, q and r, d(C_(p)−C_(q)) is a distance betweenthe camera positions, H(⋅) is a function providing a homography matrixundistorting points according to a parameter vector x_(p) and camerapositions C_(p) and C_(q), corresponding to camera p with parametervector x_(p), x_(q) and x_(r), are vectors with parameters to bedetermined so as to minimizing the error for camera p, q and r,m_(p,q)=[m^((p)) _(p,q) m^((q)) _(p,q)] denotes a set of referencepoints wherein m^((p)) _(p,q) corresponds to the respective referencepoints in camera p, m_(p,q,r)=[m^((p)) _(p,q,r) m^((q)) _(p,q,r) m^((r))_(p,q,r)] denotes a set of reference points wherein m^((p)) _(p,q,r)corresponds to the respective reference points in camera p, the set Ωincludes all pairs of partially overlapping images; the set ψ includestriplets of partially overlapping images; such that each of the Ncameras considered in the minimization problem is included in at leastone pair of cameras contained in Ω and in at least one triplet ofcameras contained in ψ, functions u(⋅) and v(⋅) extract thehorizontal/vertical component of an image point respectively,J_(p,q)=[J^((p)) _(p,q) J^((q)) _(p,q)] and J_(p,q,r)=[J^((p)) _(p,q,r)J^((q)) _(p,q,r)] denote the Jacobi matrices for camera pairs and cameratriplets, respectively, J^((p)) _(p,q) and J^((p)) _(p,q,r) denotesub-matrices corresponding to a camera p, b_(p,q) and b_(p,q,r) denoteresidual elements.

Another embodiment may have a camera system including at least onecamera being configured to provide a plurality of images from acorresponding plurality of camera positions and including a memoryhaving stored thereon calibration information derived from calibrationdata generated from an apparatus according to one of previous claims,wherein the calibration information is the calibration data orincorporates at least part thereof.

According to another embodiment, a method for obtaining calibration datamay have the steps of: receiving a plurality of partially overlappingimages of an object from a corresponding plurality of camera positionsbeing arranged along a first and a second direction according to atwo-dimensional or three-dimensional camera pattern, wherein patternshaving camera positions that differ in two directions aretwo-dimensional patterns and patterns having camera positions thatdiffer in three directions are three-dimensional patterns; selecting atleast one corresponding reference point in an overlap area of a set ofoverlapping images, and determining a displacement information along thefirst and the second direction of the reference point in each of theother images of the set of images, such that a misalignment of theplurality of images along the first and the second direction iscompensated by the displacement information so as to obtain alignedimages; determining an offset information between principal points atthe plurality of camera positions using at least three aligned images;and providing calibration data based on the displacement information andbased on the offset information, the calibration data allowing forcalibrating the plurality of images so as to comply to the camerapattern; and determining the displacement information minimizing anerror of a first minimization criteria, wherein for a 2D camera patternthe first minimization criteria is based on the determination rule;

$\mspace{11mu} {\underset{x_{1},{x_{2}x_{3}},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{{{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}}}}}$     with  x = ∑_(l)x_(l)²

determining the offset information minimizing an error of a secondminimization criteria, wherein for the 2D camera pattern the secondminimization criteria is based on the determination rule

${\; {\underset{x_{1},{x_{2}x_{3}},\ldots,x_{N}}{\arg \; \min}\sum\limits_{{({p,{q.r}})} \in \Psi}}\quad}{{{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(r)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}}}$     with  x = ∑_(l)x_(l)²

wherein argmin denotes the minimization criteria and p, q and r denoteindices of individual cameras. C_(p), C_(q), C_(r) denote camerapositions of cameras p, q and r, d(C_(p)−C_(q)) is a distance betweenthe camera positions, x_(p), x_(q) and x_(r) are vectors with parametersto be determined so as to minimizing the error for camera p, q and r,H(⋅) is a function providing a homography matrix undistorting pointsaccording to a parameter vector x_(p) and camera positions C_(p) andC_(q), m_(p,q)=[m^((p)) _(p,q) m^((q)) _(p,q)] denotes a set ofreference points wherein m^((p)) _(p,q) corresponds to the respectivereference points in camera p, m_(p,q,r)=[m^((p)) _(p,q,r) m^((q))_(p,q,r) m^((r)) _(p,q,r)] denotes a set of reference points whereinm^((p)) _(p,q,r) corresponds to the respective reference points incamera p, Functions u(⋅) and v(⋅) extract the horizontal/verticalcomponent of an image point respectively, the set Ω includes all pairsof partially overlapping images; the set ψ includes triplets ofpartially overlapping images; such that each of the N cameras consideredin the minimization problem is included in at least one pair of camerascontained in Ω and at least in one triplet of cameras contained in ψ;and/or determining the displacement information minimizing an error of alinearized first minimization criteria, wherein for a 2D camera patternthe linearized first minimization criteria is based on the determinationrule;

$\mspace{79mu} {{\underset{x_{1},{x_{2}x_{3}},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}}}} + b_{p,q}}$$\mspace{79mu} {{{wherein}\left( {{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}} \right)}^{2} \approx {\quad{{\left\lbrack J_{p,q}^{(p)} \middle| J_{p,q}^{(q)} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}} + b_{p,q}}}}$

determining the offset information minimizing an error of a linearizedsecond minimization criteria, wherein for the 2D camera pattern thelinearized second minimization criteria is based on the determinationrule

$\mspace{79mu} {{\underset{x_{1},{x_{2}x_{3}},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{\begin{bmatrix}J_{p,q,r}^{(p)} & \begin{matrix}J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{matrix}\end{bmatrix} \cdot \begin{bmatrix}x_{p}^{T} \\\begin{matrix}x_{q}^{T} \\x_{r}^{T}\end{matrix}\end{bmatrix}}}} + b_{p,q,r}}$$\mspace{79mu} {{{wherein}\left( {{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(r)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}} \right)}^{2} \approx {\quad{{\begin{bmatrix}J_{p,q,r}^{(p)} & \begin{matrix}J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{matrix}\end{bmatrix} \cdot \begin{bmatrix}x_{p}^{T} \\\begin{matrix}x_{q}^{T} \\x_{r}^{T}\end{matrix}\end{bmatrix}} + b_{p,q,r}}}}$

wherein argmin denotes the minimization criteria p, q and r denoteindices of individual cameras, C_(p), C_(q), C_(r) denote camerapositions of cameras p, q and r, d(C_(p)−C_(q)) is a distance betweenthe camera positions, H(⋅) is a function providing a homography matrixundistorting points according to a parameter vector x_(p) and camerapositions C_(p) and C_(q), corresponding to camera p with parametervector x_(p), x_(q) and x_(r), are vectors with parameters to bedetermined so as to minimizing the error for camera p, q and r,m_(p,q)=[m^((p)) _(p,q) m^((q)) _(p,q)] denotes a set of referencepoints wherein m^((p)) _(p,q) corresponds to the respective referencepoints in camera p, m_(p,q,r)=[m^((p)) _(p,q,r) m^((q)) _(p,q,r) m^((r))_(p,q,r)] denotes a set of reference points wherein m^((p)) _(p,q,r)corresponds to the respective reference points in camera p, the set Ωincludes all pairs of partially overlapping images; the set ψ includestriplets of partially overlapping images; such that each of the Ncameras considered in the minimization problem is included in at leastone pair of cameras contained in Ω and in at least one triplet ofcameras contained in ψ, functions u(⋅) and v(⋅) extract thehorizontal/vertical component of an image point respectively,J_(p,q)=[J^((p)) _(p,q) J^((q)) _(p,q)] and J_(p,q,r)=[J^((p)) _(p,q,r)J^((q)) _(p,q,r) J^((r)) _(p,q,r)] denote the Jacobi matrices for camerapairs and camera triplets, respectively, J^((p)) _(p,q) and J^((p))_(p,q,r) denote sub-matrices corresponding to a camera p, b_(p,q) andb_(p,q,r) denote residual elements.

Another embodiment may have a non-transitory digital storage mediumhaving a computer program stored thereon to perform the inventive methodfor obtaining calibration data, when said computer program is run by acomputer.

A recognition of the invention consists of having detected that, bydetermining displacement information that allows for aligning imagesalong a first direction and along a second direction and by determiningoffset information between principal points of cameras of the camerapattern, the offset information may be determined with respect topre-aligned images such that it is sufficient to determine the offsetinformation along one of the two directions such that the computationaleffort for determining the offset information along the second directionmay be saved which allows for computational efficiency. This furtherallows for a high precision and reliability of obtained imageinformation such as offset information in 3D picture by having at leasttwo independent, but correlating measurements.

Embodiments provide an apparatus that comprises a first interface forreceiving a plurality of partially overlapping images of an object froma corresponding plurality of camera positions imaged by at least onecamera, the camera positions being arranged along a first and a seconddirection and being arranged according to a camera pattern. Theapparatus comprises an analyzing unit configured for selecting at leastone corresponding reference point in an overlap area of a set ofoverlapping images, and for determining displacement information alongthe first and the second direction of the reference point in each of theother images of the set of images, i.e., a displacement of the referencepoint in the different images. A misalignment of the plurality of imagesalong the first and the second direction is compensated for by thedisplacement information so as to obtain aligned images. The apparatuscomprises a determining unit configured for determining offsetinformation between principal points at the plurality of camerapositions, using at least three aligned images. The apparatus furthercomprises a second interface for providing calibration data based on thedisplacement information and based on the offset information, thecalibration data allowing for calibrating the plurality of images so asto comply to the camera pattern. In a multi-camera system that comprisesmore than two cameras, more stereoscopic views may be obtained such asbetween a first and a second camera and between a second and a thirdcamera. Those two stereoscopic views may be brought into consistencybased on the offset information. Based on the aligned images, it issufficient to determine the offset information along the first or seconddirection which may be advantageous for at least three cameras arrangedin a single row but which is especially advantageous for atwo-dimensional or even three-dimensional camera pattern. This furtherallows for a high precision and reliability of obtained imageinformation such as offset information in 3D picture by having at leasttwo independent, but correlating measurements.

According to an embodiment, the analyzing unit is configured todetermine the displacement information using a set of parametersindicating a real or actual condition of the at least one camera at theplurality of camera positions, the parameters comprising a non-linearrelationship with respect to each other. The analyzing unit isconfigured to use a linearized version of the set of parameters and todetermine the displacement information by minimizing an error of thelinearized version with respect to a desired, e.g., error-free conditionof the plurality of camera positions. Linearization allows expansion ofa low amount of computational power. Based on the linearization, thedisplacement information along the first and second direction may,furthermore, be obtained one after the other that allows for a highrobustness of the indetermination.

According to an embodiment, the analyzing unit is configured toiteratively minimize the error of the linearized version. This allowsfor a stepwise reduction and, therefore, for an efficient determination.

According to an embodiment, the analyzing unit is configured to minimizethe error based on a determination rule that is based on a Jacobianmatrix system that corresponds to a vector of constant values when beingmultiplied with the linearized version of the parameters. This allowsfor a further reduction of computational effort.

According to an embodiment, the analyzing unit is configured todetermine the displacement information pair-wise for a pair of theplurality of images. This allows obtaining the displacement informationpair-wise and, thus, while considering a low amount of parameters. Thismay be done repeatedly so as to obtain the displacement information ofone image with respect to a different image.

According to an embodiment, the analyzing unit is configured to firstdetermine the displacement information along one of the first and thesecond direction and to subsequently determine the displacementinformation for the other direction. This allows for a high robustnessof the result against computational errors and non-linearity as only asubset of unknown values is determined at an instance.

According to an embodiment, the analyzing unit is configured todetermine the displacement information along the first directionindependent from the displacement information along the seconddirection. This allows for a small subset of unknown variables that maybe determined in a robust manner as errors along one direction may haveno or low influence on determinations of the other direction.

According to an embodiment, the calibration data is based on angles thatdescribe a rotation of the cameras, to a focal length of the camera andto a principal point of the cameras. The calibration data does notcontain information indicating the plurality of camera positions. Thisallows for a low computational effort as the positions of the camerasmay be considered as being constant and may thus remain uncomputed asvariables.

According to an embodiment, the apparatus further comprises at least onecamera being configured to provide the plurality of images from theplurality of camera positions. The apparatus is configured to determinea depth map of an object region comprising a plurality of sub regions.The at least one camera is configured to provide the corresponding imageby projecting one of the plurality of sub regions. This allows for aself-calibration.

According to an embodiment, the apparatus is configured to apply thecalibration data to the plurality of images so as to obtain a pluralityof rectified images and to provide the plurality of rectified images.This allows for precise images.

According to an embodiment, a camera system comprises a at least onecamera to provide images from the plurality of camera positions andcomprises a memory having stored thereon calibration information derivedfrom calibration data generated from an apparatus according to one ofthe previous claims. This allows for rectifying the images obtained fromthe plurality of camera positions even if the calibration data is notdetermined for an actual image but has been pre-determined.

According to an embodiment, the camera system comprises an apparatus forproviding calibration data. This allows for an apparatus that is able toself-calibrate.

According to an embodiment, a method for obtaining calibration datacomprises receiving a plurality of partially overlapping images of anobject from a corresponding plurality of camera positions being arrangedalong a first and a second direction according to a camera pattern. Themethod comprises selecting a reference point in an overlap area of oneof the images and determining displacement information along the firstand the second direction of the reference point in each of the otherimages, such that a misalignment of the plurality of images along thefirst and the second direction is compensated by the displacementinformation so as to obtain aligned images. The method comprisesdetermining offset information between principal points of the at leastone camera at the plurality of camera positions using at least threealigned images. The method comprises providing calibration data based onthe displacement information and based on the offset information.

According to an embodiment, a non-transitory storage medium has storedthereon a computer program having a program code for performing, whenrunning on a computer, a method according to embodiments describedherein.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will be detailed subsequentlyreferring to the appended drawings, in which:

FIG. 1 is a schematic block diagram of an apparatus according to anembodiment.

FIG. 2a is a schematic top view of a plurality of cameras beforemechanical calibration;

FIG. 2b is a schematic top view of the cameras of FIG. 2a aftermechanical calibration;

FIG. 2c is a schematic diagram that illustrates parameters ofembodiments described herein;

FIG. 2d is a schematic perspective view of a camera that may be used inembodiments;

FIG. 3a is a schematic diagram of a camera pattern according to anembodiment;

FIG. 3b is a schematic view of picture positions of the pattern that hasbeen described in connection with FIG. 3 a;

FIG. 3c is a schematic diagram of transformed images according to FIG. 3b;

FIG. 4a is a schematic perspective view of nine cameras that arearranged according to an embodiment;

FIG. 4b is a schematic view of example images arranged according to thepattern of FIG. 4 a;

FIG. 4c is a schematic perspective view of an example camera array thatis arranged according to a 2×8 pattern;

FIG. 4d is a schematic representation of an example 3D camera pattern;

FIG. 4e is a schematic top view of a part of the pattern of FIG. 4 d;

FIG. 5 is a schematic flow chart of method for obtaining calibrationdata according to an embodiment;

FIG. 6a is a schematic flow chart of another method for obtainingcalibration data according to a further embodiment;

FIG. 6b is a schematic diagram of a block of FIG. 6a , that illustratesa linearization strategy that may be implemented by a analyzing unitaccording to an embodiment;

FIG. 6c is a schematic representation of a graph that representstriple-matching correspondence points according to an embodiment;

FIG. 6d is a graph similar to FIG. 6c in an incomplete state;

FIG. 7 is a schematic block diagram of an apparatus according to anembodiment that comprises a plurality of cameras; and

FIG. 8 is a schematic block diagram of a camera system according to anembodiment that comprises a plurality of cameras and comprises a memoryhaving stored thereon calibration information.

DETAILED DESCRIPTION OF THE INVENTION

Equal or equivalent elements or elements with equal or equivalentfunctionality are denoted in the following description by equal orequivalent reference numerals even if occurring in different figures.

In the following description, a plurality of details is set forth toprovide a more thorough explanation of embodiments of the presentinvention. However, it will be apparent to those skilled in the art thatembodiments of the present invention may be practiced without thesespecific details. In other instances, well known structures and devicesare shown in block diagram form rather than in detail in order to avoidobscuring embodiments of the present invention. In addition, features ofthe different embodiments described hereinafter may be combined witheach other, unless specifically noted otherwise.

In the following, reference is made to camera arrays comprising aplurality of cameras that are arranged at different positions. Sucharrays are described herein as being arranged according to atwo-dimensional pattern for illustrating the advantages of theembodiments. Although referring to a two-dimensional pattern, otherpatterns may also be used that comprise cameras being arranged accordingto a three-dimensional pattern. For example, this may be considered whenallowing for divergences of the focal length of a camera. According toother embodiments, the camera pattern may comprise only one dimension,i.e., the cameras are arranged along a single line (or row). Someembodiments are described as one camera being arranged at each cameraposition such that embodiments that relate to a plurality of partiallyoverlapping images of an object being received from a correspondingplurality of camera positions may be understood as the plurality ofimages being received from a corresponding plurality of cameras.

Without limitation one camera may capture two or more images atdifferent locations when being moved from a location to another suchthat images from different camera positions may be captured by a singlecamera.

Embodiments described herein are explained in terms of stepwiserectifying images. This may relate to rectify or transform images so asto obtain transformed images and to use such transformed images forfurther steps or for further processing. Without any limitation, suchdescriptions rate to correction data that is obtained stepwise, i.e.,the original images may remain partially or completely unchanged,wherein a transformation of the images is only theoretically performed,e.g., when determining displacement data. Thus, rectifying ortransforming the images may be equivalent to determining the amount ofthe transformation that has to be applied to the images.

Embodiments described herein relate to a displacement of images withrespect to each other and/or with respect to a camera pattern. Thedisplacement relates to a transformation according to a first and/orsecond direction parallel to an object plane but also relates to arotation of the image in the three-dimensional space. In the following,reference is made to rectifying images, i.e., to calibration of images.Such rectification may comprise a transformation of images. Suchtransformation may relate to a shifting or a movement of the image alongat least one image direction, to a rotation of the image in the imageplane and/or to a scaling (skew) of images. Thus and unless statedotherwise, reference made herein that relates to a shift of an imagealso relates to a rotation and/or a scaling of the image.

FIG. 1 is a schematic block diagram of an apparatus 10 according to anembodiment. The apparatus 10 comprises an input interface 12 forreceiving a plurality of partially overlapping images of an object orobject area 16. The images 14 are provided from a correspondingplurality of cameras 18 a to 18 c that project the object 16 byprojecting sub areas 22 a to 22 c of the object 16 wherein each sub area22 a to 22 c may correspond to one of the plurality of images 14. Anoverlap of the sub regions or sub areas 22 a to 22 c may correspond toan image overlap of the images 14. The plurality of cameras 18 a to 18 cis arranged according to a pattern along a first direction x and along asecond direction y, for example, in an x/y plane. The apparatus 10comprises an analyzing unit 24 configured for selecting one or morereference points 26 in an overlap area 28 of one of the images 14 whichis illustrated, by non-limiting example only, for reference point 26 ₁in an image 14 a of the plurality of images 14. The image 14 a overlapswith another image 14 b of the plurality of images 14 in the overlaparea 28, i.e., in the overlap area 28, both images 14 a and 14 b showthe same part of the object 16. The reference point 26 contained asreference point 26 ₁ in the image 14 a is also contained as referencepoint 26 ₂ in the image 14 b. For example, the image 14 a is captured bycamera 18 a and the image 14 b is captured by camera 18 b. Due to amisalignment between the cameras 18 a and 18 b, the reference points 26₁ and 26 ₂ are displaced by a displacement 32 along the x directionand/or y direction. The analyzing unit 24 is configured to determine thedisplacement information 34 that indicates the displacement 32 along thex direction and the y direction. The misalignment of the images 14 a and14 b along the x direction and the y direction is compensated byapplying the displacement information 34 which allows obtaining alignedimages. Determining the displacement information 34 so as to obtaininformation on how to transform the images and allowing for low or aminimum of remaining errors for all of the considered images may be afirst optimization problem. The analyzing unit 24 may be configured tosolve the optimization problem, as will be described later.

Further, the analyzing unit is configured for receiving the images 14from the plurality of cameras, e.g., without or only coarse priorknowledge with respect to the content thereof and/or overlaps of theimages. The analyzing unit may be configured for selecting acorresponding reference point, i.e., image content in the overlap area,e.g., by comparing images. The corresponding reference point orreference region may be a single point or region/object in the capturedscene. By identifying similar or identical content, the overlap area maybe identified and a set of cameras (pairs, triplets, etc.) may beidentified having overlaps in their images at least at a regioncomprising the reference point 26 ₁ and/or 26 ₂. The set of images maythus be configured such that the set of images is a subset of theplurality of images 14, i.e., at least one of the images is notcontained in the respective set. In contrast each camera is contained inone or more, e.g., at least two sets.

Based thereon relationships between the cameras may be determined asbeing described in connection with FIG. 6c . The analyzing unit 24 maybe configured for determining offsets or shifts or other deviations ofthe reference point starting from one image in one (pair of images) tow(triplet of image) or a higher number of other images. The analyzingunit 24 may be configured for determining further reference points forthe same set of cameras and/or for other sets of cameras.

Further images of the plurality of images 14 may be misaligned withrespect to the image 14 a and/or the image 14 b. The analyzing unit 24is configured to determine the displacement, to determine thedisplacement information 34, respectively, between the plurality ofimages 14. The analyzing unit 24 may be configured to determine themisalignment or displacement globally, i.e., to select all images of theplurality of images 14 that illustrate the same reference point 26 andto determine the displacement information globally for the selected setof images. Alternatively, the analyzing unit 24 may be configured toalign two images such as the images 14 a and 14 b with respect to eachother, i.e., to operate pair-wise. The analyzing unit 24 may beconfigured to select a set of images that comprise the same referencepoint 26 and to pair-wise align the images with respect to each other.The analyzing unit 24 may be configured to select more than just onereference point in the overlap area such as, for example, at least 5,10, or 100. For determining the displacement information along one ofthe first and second direction, a first image of a selected pair ofimages may be rotated and scaled so as to be aligned along one imagedirection, i.e., such as lines or columns of pixels point towards theother camera position. Then, the second image may be transformed alongthe respective direction so as to align the at least one selectedreference point. This may be repeated for the other direction. Themachine has knowledge about the position of all cameras. It is evenassumed that this information is reliable.

The apparatus 10 comprises a determining unit 36 configured fordetermining a offset information 38 that relates to principal points 42a to 42 c of the plurality of images 14. The offset information may thusbe a distance information that describes a distance between principalpoints and may thus be a relative information relative with respect tothe images associated to the principal points. Based on a physicaldistance between cameras (or camera positions) 18 a and 18 b or betweencameras (or camera positions) 18 a and 18 c or between cameras (orcamera positions) 18 b and 18 c, a view on the scene or object area,i.e., the object 16 is different from camera to camera. This effect isknown as disparity and is used for determining 3D images. For astereoscopic view, two cameras are needed, for example, cameras 18 a and18 b. Through the use of camera 18 b and 18 c, 18 a and 18 c,respectively, a second stereoscopic view may be obtained that may allowfor an increase in information. When considering now a firststereoscopic view being obtained by using cameras 18 a and 18 b and asecond stereoscopic view being obtained by using cameras 18 b and 18 c,the stereoscopic information may differ from one another when distances44 a between principal points 42 a and 42 b and a distance 44 b betweenthe focal points 42 b and 42 c differ from each other. Based thereondisparity estimation and/or depth reconstruction might lead to differentresults, which may be prevented by the described embodiments.

The determining unit 36 is configured to determine the offsetinformation 38 so as to comprise information related to the distances 44a and 44 b. The determining unit 36 is configured to use at least threeimages, for example, the images of the cameras 18 a to 18 c, fordetermining the offset information 38. For determining the distances 44a and/or 4 b the reference points may be used. The respective referencepoint may be the identical in a pair of images and may be identical ordifferent from each other in different sets of images, i.e., in apicture that is part of different pairs of images, different referencepoints may be used. Determining the offset information 38 so as toobtain information on how to transform the images and allowing for lowor a minimum of remaining disparity offset for all of the consideredimages may be a second optimization problem. The determining unit 36 maybe configured to solve the optimization problem, as will be describedlater.

The apparatus 10 comprises an interface 46 for providing calibrationdata 48 that is based on the displacement information 34 and based onthe offset information 38. The calibration data may allow for rectifyingthe images 14 including a compensation in differences between distances44 a and 44 b between the focal points 42 a to 42 c. The apparatus 10may comprise an optional directing unit 52 being configured to processand/or to direct the calibration data 48. For example, the apparatus 10may comprise an optional memory 54, wherein the directing unit 52 isconfigured to store the calibration data 38 in the memory 54. Thedirecting unit 52 may be implemented by a central processing unit (CPU),by a microcontroller, by a field programmable gate array (FPGA) or thelike.

Based on the displacement information 34, the apparatus 10 has knowledgeabout the displacement of the images 14 and may compensate for thedisplacement either by transforming such as shifting the images withrespect to each other or by considering the displacement during furtheroperation. This allows determining the differences in the distances 44 aand 44 b using aligned images such that the distance may sufficiently becalculated along one direction such as x for obtaining sufficient,precise information. An alignment of the images 14 may allow forreconstructing one dimensional dependencies in the camera pattern suchas lines, rows or diagonals of the cameras 18 a to 18 c. The alignmentperformed by the analyzing unit 24 may, for example, allow for each ofthe images 14 being in-line or in-row of the camera pattern. This mayleave unconsidered a distance between two lines of the pattern, or tworows of the pattern, respectively. However, by determining the distance44 a and the distance 44 b, differences in the disparity betweendifferent stereoscopic images may be compensated for. Thus, bypre-aligning the images, the compensation for deviations in distances ofprincipal points may be performed along one direction and may, at thesame time, provide sufficient information. Thus, disparity estimationand/or depth reconstruction as well as rectification of images may beperformed with low computational effort.

FIG. 2a shows a schematic top view of the cameras 18 a to 18 c beforemechanical calibration. Optical axes 54 a to 54 c of the cameras 18 a to18 c differ from each other.

FIG. 2b shows a schematic top view of the cameras 18 a to 18 c aftermechanical calibration. The optical axes 54 a to 54 c are essentially inparallel but may differ from each other by the displacement described inconnection with FIG. 1. In addition, distances b_(x12) between aposition C₁ of camera 18 a and position C₂ of camera 18 b may bedifferent when compared to a distance b_(x23) between the position C₂and the position C₃ of camera 18 c, wherein the distances b_(x12) andb_(x23) relate to the x direction. C₁, C₂ and C₃ are positions of thecameras 18 a, 18 b and 18 c in the three-dimensional world.

In other words, FIGS. 2a and 2b show the top view of an array from thetop before and after (mechanical) calibration. Before the calibrationorientation of cameras C₁, C₂ and C₃ are not consistent and not inparallel. The calibration rotates the cameras such that their opticalaxes are parallel and perpendicular to the base line of the cameras.Thus, FIG. 2a shows a top view of the camera array before calibration.The optical axes of the cameras are slightly different. FIG. 2b shows atop view of the camera array after calibration. The optical axes are inparallel. Both FIGS. 2a and 2b may show only a top row of the cameraarray.

FIG. 2c shows a schematic diagram that illustrates parameters ofembodiments described herein. The three dimensional world may bedescribed by the x direction, the y direction and the z direction thatmay be arranged perpendicular to each other. A rotation by an angle αaround the x direction may be referred to as “tilt”. A rotation by anangle β around the y direction may be referred to as “yaw”. A rotationby an angle γ around the z direction may be referred to as “roll”. Theillustrated parameters may be part of a vector that is optimized forobtaining rectified images.

FIG. 2d shows a schematic perspective view of a camera 18 that may bearranged in the array of cameras and may thus be camera 18 a, 18 b or 18c. The camera 18 comprises and image sensor 58 that comprises a centerpoint 62. Properties of an imaging lens 64 or an array thereof maydetermine or at least influence a focal length f of camera 18. The focallength f may face an error at. Furthermore, the location of the lens 64may at least influence the location of the principal point 42 on theimage sensor 58 that may be different from the center 62. The principalpoint 42 may describe a center point of the image that is projected,wherein the point 62 may be a center point of the image sensor 58. Adivergence or distance between the projected sub region of the objectarea and the center of the image sensor 58 may be described by adistance p_(x) along the x direction and by a distance p_(y) along the ydirection. The parameters p_(x) and p_(y) may be assumed to be almostzero for calibrated cameras so as to denote that the optical center ofthe camera coincides with the center 62 of the imaging sensor.

When referring again to the model of the pinhole camera, the projectionmatrix P may be formulated according to:

P=K·R·[I|−C]  Formula 1

K is, for example, a 3×3 matrix consisting of the intrinsic parameters.These include the focal length, the principal point τ and the skewfactor s.

FIG. 3a shows a schematic diagram of an example of a camera pattern 30according to which cameras 18 a to 18 d are arranged, i.e., the pattern30 is a 2×2 pattern in the x/y plane. The camera array may be in a statethat may be obtained after a mechanical calibration that is explained inconnection with FIG. 2b . Camera 18 a may be arranged at position C₁,camera 18 b may be arranged at position C₂, camera 18 c may be arrangedat position C₃, and camera 18 d may be arranged at position C₄, thatare, in an error-free state, arranged according to a rectangle or asquare. Other numbers of cameras and/or other patterns are possible.

Parameters that may be evaluated for determining the displacementbetween images may be a focal length. In FIG. 3a , the focal lengtha_(f1) of camera 18 a is different from the focal length a_(f3) ofcamera 18 c which results in different sizes 66 ₁ and 66 ₃ along the xdirection and/or different sizes 68 ₁ and 68 ₃ along they direction ofpictures 14 a and 14 c. Furthermore, picture 14 b may be rotated by anangle γ₂. Thus, the displacement information may relate to atransformation such as a shift, a rotation and/or a scaling of theimages 14 a to 14 d along lateral directions x, y and/or z as well as toa rotation of images around at least one axis as described in connectionwith FIG. 2 c.

FIG. 3b shows a schematic view of pictures 14 a to 14 d of the 2×2pattern that has been described in connection with FIG. 3a . On the lefthand side, the images 14 a to 14 d are displaced; in particular, theimage 14 d does not match the intended pattern. On the right hand side ascenario is illustrated that may be obtained when compensating for thedisplacement. Arrows 72 ₁ to 72 ₃ indicate a direction along which theimages 14 a to 14 c have to be transformed including one or more of ashift, a rotation and a scaling, with respect to image 14 d such thatthe reference points 26 ₁ to 26 ₄ correspond to the camera pattern. Thearrows 72 ₁ to 72 ₃ may also indicate a distance along which therespective image 14 a to 14 c has to be transformed along the directionsx and/or y and may therefore indicate the displacement information. Theblack parts in the images 14 a to 14 d schematically illustrate regionsfrom which the image has been transformed away.

The analyzing unit 24 may be configured to determine the displacementinformation along the x direction and subsequently along the y directionor vice versa. When compared to a determining of the displacement alongx direction and y direction simultaneously, this may provide for a highrobustness. The apparatus may have knowledge about the error-freepattern 30 and may determine that lines 74 a and 74 b should be inparallel to each other and that columns 76 a and 76 b should also be inparallel to each other. The analyzing unit 24 may be configured todetermine that this is not true and that the images have to betransformed with respect to each other as illustrated on the right handside of FIG. 3b where the lines 74 a and 74 b and the columns 76 a and76 b are each in parallel with each other. The determination of thedisplacement information may also be executed independently from eachother, i.e., lines 74 a and 74 b may be brought into a parallelcondition independent of columns 76 a and 76 b or vice versa. This mayrefer to a first direction that may be corrected independent from thesecond direction. The second direction such as the columns 76 a and 76 bwhen first having transformed the lines 74 a and 74 b is executeddependent on the other direction so as to maintain the corrected images.

Thus, the left hand side of FIG. 3b shows an initial state beforecalibration but after mechanical calibration. The bottom right handimage is clearly not aligned with the other three images. On the righthand side of FIG. 3b rectified images are illustrated. Correspondingpixels are in the same row for horizontal adjacent views and the samecolumn for vertical adjacent views.

In other words, the left hand side of FIG. 3b shows a 2×2 camera arrayin a general state. Camera 2 is slightly rotated and camera 3 has aslight deviation in focal length.

FIG. 3c shows a schematic diagram of the transformed images 14 a to 14 daccording to FIG. 3b on the right hand side. Based on differentpositions C₁ to C₄ of the cameras, the structure that is projected asreference points 26 ₁ to 26 ₄ on the images 14 a to 14 d is arranged onlaterally different positions within the images 14 a to 14 d. On images14 a and 14 c, the reference point or reference structure 26 ₁ maycomprise a distance u₁ to a first side of the image, wherein distancesindicated with u may be, for example, in parallel with the x direction.In pictures 14 b and 14 d, reference points 26 ₂ and 26 ₄ may face adistance u₂ that is different from u₁ to a comparable side of the image.The same effect may relate to images 14 a and 14 b with respect to adifferent perpendicular side. The reference points 26 ₁ and 26 ₂ mayface a distance v₁ to that side, wherein the distance may be inparallel, for example, along the y direction. The reference points 26 ₃and 26 ₄ of images 14 c and 14 d may face a different distance v₃ to thecomparable side.

An angle φ₁₄ indicates an angle by which image 1 (24 a) has to berotated around an anchor point, possibly around a center 27 ₁ of theimage 24 a, such that its direction u or alternatively v at the anchoris directed towards the comparable anchor 27 ₄ in image 4 (24 d). Anangle φ₂₄ indicates an angle by which image 2 (24 b) has to be rotatedaround an anchor point, possibly around a center 27 ₂ of the image 24 b,such that its direction u at the anchor is directed towards thecomparable anchor 27 ₄ in image 4 (24 d). Such angles may provide forinformation indicating the displacement information. φ may therefore bea parameter fixed for a pair of images or views (p,q) as it only dependson the mechanical center/position of cameras p and q. Centers 24 ₁ to 24₄ may also indicate the used camera position C₁ to C₄.

FIG. 4a shows a schematic perspective view of nine cameras 18 a to 18 ithat are arranged according to a 3×3 pattern.

FIG. 4b shows a schematic view of the pattern 40, wherein the images 14a to 14 i are arranged according to the pattern 40 of the cameras 18 ato 18 i in FIG. 4a . Distances b_(x) and b_(y) may differ from camera tocamera which may be compensated for when determining the offsetinformation.

In other words, FIG. 4b shows a schematic view of a 3×3 camera arrayfrom the back side. Cameras in the array are consecutively numberedstarting at the top left corner with camera C₁ going on to the right, tothe end of the row, and then continuing on to the next row in theleft-most column and so on. The last camera in the bottom right corneris consequently denoted C_(M×N), M indicating the row an N indicatingthe column of the pattern.

FIG. 4c shows a schematic perspective view of a camera array 40′ that isarranged according to an 2×8 pattern. Any other patterns comprising anarbitrary number of cameras may be implemented. The concept may beimplemented for a number of three or more images. The at least threeimages may be acquired at different locations with at least one camera.As the rectification is done on an image-basis, also a single camera maybe used that is moved between taking the pictures. The distance betweenthe centers of the cameras or the principal points may then relate tothe distance when having moved from position one to position two.Accordingly, to be able to take at least three pictures, also twocameras may be used, wherein at least one camera is moveable. Threeimages may also be acquired by three cameras such that for any number ofimages a corresponding number of cameras or a lower number may be used.

Although having described patterns that comprise a square or rectangularshape, any other shape may be realized. Any number of cameras such as atleast three, five, nine or sixteen may be used. Thus, known approachesare extended towards two dimensional arrays of cameras, for example,such as shown in FIGS. 4a and 4c . Such two dimensional systems maycomprise a set of cameras but may also be mounted on a gantry that isable to move the camera to a specific position and take a picture.

Apparatus 10 implements a concept that transforms the images such thatthe remaining deviation compared to a given model is minimized. This maybe achieved by approximating a highly non-linear problem by a series ofsub-problems that may be of a linear nature. The overall problem may beapproximated and solved using iterative minimization methods minimizingthe residual error. Alternatively, the apparatus 10 may also beconfigured to solve a non-linear optimization.

FIG. 4d shows a schematic representation of a 3D camera pattern 40″ thatis arranged, by non-limiting example, on a curved surface such as on asphere. The pattern may also be considered to have lines 74 a-d and/orcolumns 76 a, wherein the lines and/or columns may be occupied differentfrom each other. Alternatively or in addition, the lines and columns maybe arranged in a non-perpendicular manner with respect to each other,along curved lines and/or columns or according to other rules.

FIG. 4e is a schematic top view of a part of the pattern 40″ of FIG. 4d. Objects 77 and 79 may be part of a scene to be captured. The pairs ofcameras 18 b/18 e and 18 d/18 e of line 74 c may each have differentdirection of view and different distances to the objects 79 a and 79 bthat are seen by both pairs which may lead to additional informationneeded when compared to a 2D-pattern.

In other words, typically, the precision of mechanical setups islimited. Therefore, the exact properties of one or several cameras needto be reconstructed. Such a reconstruction of camera parameters is alsoknown as calibration. In this procedure, different measurements are madeand based on a physical or mathematical model, unknown parameters arefound using the analyzing unit and/or the determining unit. Once thoseunknown parameters are found, one can use these parameters, for example,to define a function that transforms a set of non-rectified images intoa set of rectified images. Thus, the obtained calibration data may beapplied to non-rectified images which allows for storing or transmittingthe non-rectified images even if the displacement information hasalready been obtained.

The focal length f of a camera may differ from its intended/ideal valuein horizontal and/or vertical direction via the error at. The skewfactor s may be used to describe the skew of a pixel. In the case ofmodern imaging sensors, this factor is typically 0, meaning that pixelcells are rectangular. The principal point describes the point ofintersection between the camera's optical axis and the sensor. If thesensor is positioned exactly beyond the central point of the opticalaxis, τ_(x), deviation along the x direction and τ_(y), deviation alongthe y direction, both are zero. Therefore, the properties of a cameramay be formulated as a matrix as:

$\begin{matrix}{K = \begin{pmatrix}f_{x} & s & \tau_{x} \\0 & f_{y} & \tau_{y} \\0 & 0 & 1\end{pmatrix}} & {{Formula}\mspace{14mu} 2}\end{matrix}$

By convention it may be formulated that, in a default state, an idealcamera is oriented such that it is strictly pointing down the z axis ofa coordinate system such as the one illustrated in FIG. 2d . Image linesmay be parallel with the x axis while image columns may be parallel withthe y axis, respectively and by non-limiting example only. It should beunderstood that different coordinate systems may be used that may betransferred from one into another without limitation. In such a case,the rotation matrix may be the identity matrix I. This is not constantlythe case and therefore the rotation matrix R is a general rotationmatrix that is constructed from three angular components α, β and γ,denoting rotations about the primary axes of the coordinate system asdescribed in connection with FIG. 2c . The camera position C may begiven as a three dimensional coordinate. The projected points or imagepoints m_(i) from a set of 3D world points M₁ . . . M_(N) may then becomputed according to the determination rule:

$\begin{matrix}{\begin{bmatrix}m_{1}^{T} & \ldots & m_{N}^{T}\end{bmatrix} = {P{\text{|} \cdot \begin{bmatrix}M_{1}^{T} & \ldots & M_{N}^{T} \\1 & \ldots & 1\end{bmatrix}}}} & {{Formula}\mspace{14mu} 3}\end{matrix}$

In this case [M_(i) ^(T), 1]^(T), denotes a homogenous 4D coordinate.The resulting image coordinate m, may then be a three-element vector.This point represents an image coordinate such that the vector may benormalized to its third component. After this normalization, u and vdenote the horizontal/vertical position of a point in an image planewhich may also be understood as distance (in pixels) from a side of theimage.

For each vector m, the following rule may be applied:

$\begin{matrix}{m_{i} = \begin{pmatrix}u_{i} \\\begin{matrix}v_{i} \\1\end{matrix}\end{pmatrix}} & {{Formula}\mspace{14mu} 4}\end{matrix}$

In the context of multi-camera self-calibration, bundle adjust (BA), asdescribed in [2], chapter 18, and which is also known asbundle-adjustment, is a known and proven algorithm. This algorithm usesiterative optimization methods like the Levenberg-Marquardt algorithm tominimize the following problem:

$\begin{matrix}{\min\limits_{{\hat{P}}^{i},{\hat{M}}_{j}}{\sum\limits_{ij}{d\left( {{{\hat{P}}^{i} \cdot {\hat{M}}_{j}},x_{j}^{i}} \right)}}} & {{Formula}\mspace{14mu} 5}\end{matrix}$

wherein d(⋅) denotes the geometric distance between an image point x_(j)^(i) and a re-projected point {circumflex over (P)}^(i)·{circumflex over(M)}_(j)·{circumflex over (P)}^(i)·{circumflex over (P)}^(i) denotes theestimated camera matrix for camera i and {circumflex over (M)}_(j)denotes an estimated world point. Variants of this algorithm are knownthat are designed to improve efficiency for specific applications andworking points. In [3] a method is presented to calibrate a similarcamera system consisting of 25 cameras in total. A well-knownchart-based calibration method presented in [4] is used and the resultis refined using global optimization. Other works as presented in [5],[6] and [7] use additional means like projected patterns or laserpointers to calibrate a set of cameras or entail actively controllingthe camera orientation or zoom. A calibration procedure as described in[8] whilst making references to [9] involves the user to provide acalibration object (such as a checkerboard) in front of the multi-camerasystem. Furthermore the system is limited to two-dimensional (2D) camerasetups and the calibration object needs at least to be partially visiblein all cameras at once. The properties of the calibration object need tobe known to the algorithm. Thus, the calibration object cannot beselected arbitrary. Thus, in known systems, properties of thecalibration object are known to the algorithm. I.e., the number ofpatches on the checkerboard or the fact that all patches are squares.This is directly exploited in the algorithm. The algorithm will fail ifthe calibration object cannot be detected properly.

Camera systems such as the ones illustrated in FIGS. 4a and 4c andschematically illustrated in FIG. 4b are arranged so as to share onecommon plane and their optical axis may be roughly, i.e., at most ±15%,±10% or ±5% deviation is possible, perpendicular to that plane.Additionally, all cameras form a rectangular grid. A grid may beunderstood as comprising lines 74 and columns 76 that may be, but do notneed to be, perpendicular to each other. The apparatus 10 has knowledgeabout the physical layout of the camera setup, i.e., the analyzing unit24 and the determining unit 36 have such knowledge. The model orknowledge includes knowledge about a camera position C and each camera'sideal orientation. The ideal orientation is thus a specified condition.

The starting point of the calibration procedure that may be performedwith apparatus 10 may be seen on the left hand side of FIG. 3b . Itillustrates a subsystem of four cameras that is arranged on a 2×2 gridand captures a scene from slightly different positions. Correspondingimage points, i.e., the reference points 26 ₁ to 26 ₄, are connectedwith lines. A set of corresponding points denotes image points, regionsor objects that refer to the same 3D scene point. Thus, even if definingreference points, embodiments are not limited to single points but referto image regions or objects. In other words those individual imagepoints correspond to a single scene point. This point/object is observedin two or more individual cameras. Corresponding points can be detectedby means of feature detection and matching algorithms. Those algorithmsassume that the projection of an object into different cameras resultsin similar image patches. Due to different distortions such as non-exactmechanical alignment, corresponding points typically form a non-regularquadrilateral. This is not very descriptive and establishing a simplerelationship between those corresponding points 26 is intuitivelydifficult or impossible. Therefore, the images are transformed such thatthe connecting shape is rectangular for all corresponding points in theimages. By aligning the reference points 26 ₁ to 26 ₄, other selectedreference points may also be aligned. An aligned version is illustratedin FIG. 3b on the right hand side. Once this requirement is met, furtherprocessing steps like stereo-matching may be executed efficiently.

Based on the pinhole camera model and the projection matrix P, thesituation on the right hand side of FIG. 3b may be met if the cameras'focal length is identical and the optical axis of the cameras areparallel and perpendicular to the plane formed by the cameras' centers.

In the remaining part of the document subscripts as in x_(p) are used toindicate that a vector x denotes properties or information belonging toa specific element such as a camera p. If a variable encodes informationbelonging to a pair or tuple of cameras we denote this using atwo-element subscript such as φ_(pg) In this example φ_(pg) encodes anangular relationship between to cameras p and q. Similarly, relationshipor information comprising three cameras is encoded using a three elementsubscript such as in A_(pqr).

The calibration task can be formulated as measuring the deviation fromits ideal state and finding a set of functions H(x_(p),C_(p),C_(q)) andH(x_(q), C_(p), C_(q)) that, given a pair of images, transforms eachimage point such that corresponding image points are located in the sameimage row. For example, matrix H is a 3×3 homography matrix.Mathematically, this may be expressed as:

                                  Formula  6v(H(x_(p), C_(p), C_(q)) ⋅ P_(p) ⋅ M) − v(H(x_(q), C_(p), C_(q)) ⋅ P_(q) ⋅ M) = 0                                  Formula  7${\underset{{left}\mspace{14mu} {view}}{\underset{}{v\left( {H\text{|}{\left( {x_{p},C_{p},C_{q}} \right) \cdot m_{p}}} \right)}} - \underset{{right}\mspace{14mu} {view}}{\underset{}{v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{q}} \right)}}} = 0$

M denotes a 3D world coordinate, P_(p) denotes the projection matrix forcamera p. Typically, M as well as P are unknown. Assuming that the worldpoint M can be seen in both cameras p and q, this point forms one pairof corresponding image points m_(p) and m_(q). H(x_(p),C_(p),C_(q)) is afunction of each camera's position C_(p) and C_(g) and several intrinsicparameters x_(p). The row vector x may be formulated according to therule:

x:=[αβγ_(f) p _(x) p _(y)κ]   Formula 8

H(x_(p),C_(p),C_(q)) may be modeled using the pinhole camera model. Theouter function v(⋅) is used to extract the vertical component of animage point. The equation given above may be implemented for allpossible pairs of cameras for a given subset which is relevant for thefurther processing. Although Formulas 6 and 7 and other descriptionsmade herein differentiate between left view and right view, this may bechanged with respect to each other or manipulated, without anylimitation. When referring, for example, to FIG. 3b, x direction and ydirection are illustrated as travelling from left to right or frombottom to top and therefore define positive and negative values ofFormulas 6 and 7. By redefining the coordinate system such as whenletting x travel from right to left, the left view and the right vieware interchangeable. It is therefore important to differentiate betweenthe views of a stereo setup in view of the coordinate system. Forexample, p may be related to the left view and q may be related to theright view or camera. In other words, camera q is to the right of camerap. For this reason, the ordering of the camera positions C is p, q inthe left part of Formula 7 as well as in its right part.

The problem discussed above is typically valid for the case of a pinholecamera that does not suffer from side effects like lens-distortion. Iflens-distortion is relevant for a specific camera and lens in animplementation Formula 7 may be extended:

v(H(x _(p) ,C _(p) ,C _(q))·undist(m _(p) ,x _(p)))−v(H(x _(q) ,C _(p),C _(q))·undist(m _(p) ,x _(q)))   Formula 9

Undist( ) denotes a function that reverts lens-distortion according tosome model and parameters contained in x as κ (Formula 8). Undist( ) maybe defined as:

$\begin{matrix}{{{undist}\left( {m,x} \right)}:=\begin{pmatrix}{\tau_{x} + {{L\left( {\sqrt{u^{2} + v^{2}},\kappa} \right)} \cdot \left( {u - \tau_{x}} \right)}} \\{\tau_{y} + {{L\left( {\sqrt{u^{2} + v^{2}},\kappa} \right)} \cdot \left( {v - \tau_{y}} \right)}} \\1\end{pmatrix}} & {{Formula}\mspace{14mu} 10} \\{{L\left( {r,\kappa} \right)}:={1 + {k_{1} \cdot r} + {k_{2} \cdot r^{2}} + \cdots}} & {{Formula}\mspace{14mu} 11}\end{matrix}$

Function L(r, κ) actually models the distortion and is controlled by κ.Different models may be selected as well as a different length ofparameter vector κ. Here, the discussion is limited to two elements inκ. In the ongoing part of this text, explicit elements in κ will bereferred to as k such that κ may be expressed as: κ=[k₁, k₂, . . . ].

In principle, this problem may be solved using stereo calibrationmethods on each pair of cameras. In general, such a method will estimatea parameter set x_(p), x_(q) for images p and q, but a differentparameter set {tilde over (x)}_(p),{tilde over (x)}_(r) for images p andr, wherein image r is located, for example, below picture p such asimage 14 c with respect to image 14 a. In contrast, it is an object toobtain only one set of parameters x per camera. The desired calibratedsituation may also be expressed using the camera matrices P. In thecalibrated case, all cameras have identical camera matrices K. Allcameras are orientated identical with their optical axis perpendicularto the camera baseline. In this situation, the camera matrix P onlydepends on the camera position C as described by:

P _(p) =K·[I|−C _(p)]  Formula 12

In the case of independent stereo calibration, the obtained solution isslightly different and doesn't match all criteria: Self-calibrationmethods can only estimate a relative difference between two cameras.This is, of course, also true for the principal point. In case of ahorizontal calibrated stereo rig with corresponding image points onidentical image lines, both horizontal principal points τ_(x,1) andτ_(x,2) may be selected and manipulated arbitrarily. Both images stillform a stereo system. This fact is, in example, also exploited instereoscopic productions to manipulate the convergence plane.

In case of complex image processing such as multi-camera disparityestimation, one likes to ensure that corresponding image points acrossmore than two views can be correlated by a simple relationship. Based onthe simplified projection matrices given in Formula 9, one can deducethe following formula:

$\begin{matrix}{{\frac{{disp}_{1,2}}{{C_{2} - C_{1}}} - \frac{{disp}_{1,3}}{{C_{3} - C_{1}}}} = 0} & {{Formula}\mspace{14mu} 13}\end{matrix}$

Given a triplet of matching feature points, one can see that stereodisparity directly corresponds to the distance between the cameras'centers. The denominator of Formula 10 may be the distance 44 or b_(x),b_(y), respectively.

In a real setup, true camera positions will slightly differ from thegiven values C and known calibration methods are in general notapplicable. However, the concept according to embodiments describedherein assumes that the positional error is very small and can beignored which allows for a reduced space of parameters and therefore fora lower complexity of the problem. The concept according to theembodiments described herein may be applied but may slightly increaseback-propagation error.

It is therefore one aspect of the embodiments described herein to firstalign the images and afterwards determine the distance between theprincipal points by using a different number of images. For determiningthe displacement information, images may be used pair-wise, wherein theoffset information may be determined using three cameras.

The displacement information as well as the offset information may bedetermined without user-interaction. For each pair of images/triplet ofimages robust corresponding points, i.e., reference points, regions orobjects are determined in an automatic manner, especially without a userproviding special calibration patterns such as checkerboards or similarobjects. In contrast to existing systems (such as [8]) it is in additionnot needed that a single (possibly known) object or point can beobserved in all cameras at one time.

Thus, the number of cameras is adapted to the problem to be solved.Furthermore, the underlying non-linear optimization problem may bereformulated. The analyzing unit and/or the determining unit may beconfigured to solve the non-linear optimization problems. According toan embodiment, the analyzing unit and/or the determining unit may beconfigured to linearize the non-linear problem for some distinct pointsand to solve the linearized versions using standard iterativeminimization algorithms. Instead of solving one large non-linearproblem, such embodiments set up a list of local, piece-wise linearproblems that are used to formulate the global problem. This globalproblem may then be solved iteratively using linear methods. Whenassuming that the mechanical design of the system is known, includingcamera position and ideal orientation, the calibration procedure mayenforce camera views to obey this model. Additionally, the determinationof the displacement information may be split into two parts. By this,highly correlated parameters leading to unstable results can beseparated and optimized sequentially. Known approaches such as BAinvolve estimating 3D world points from image correspondences. This is,in itself, a complex, erroneous procedure. In embodiments, 3D worldpoints are not needed, increasing the stability and efficiency of theapproach. By example, BA optimizes camera parameters as well as positionof 3D world points. The total number of degrees-of-freedom can becomputed as 7×N+3×P. In this case, N denotes the number of cameras whileP denotes the number of 3D world points. According to embodiments, afocus is set on the intrinsic parameters and the camera rotation. This,initially, yields, in an example, number of degrees-of-freedom of 8×N.When referring again to Formula 8, the vector x comprises a number of 6parameters. When formulating a problem based on the 6 parameters thatrely on a single camera, the number of cameras N×8 values per camerayield in the value above. Thus, embodiments described herein focus onobtaining a result, i.e., the displacement information and the offsetinformation, that makes subsequent image processing steps efficient. Inthe first step, correspondences between image pairs are sufficient. Thesecond step involves correspondences in three, possibly adjacent, views.This solution may spread an error in the camera array among all cameras,which is a specific property of embodiments described herein. Thus, anapparatus such as apparatus 10 may be configured to impinge on anerror-free image with an error so as to compensate for inter-imageerrors. The concept is designed to fit a set of images to a given model,i.e., the camera pattern. The physical model, as well as the mappingbetween an image and its position in the array, is specified, by a user.

FIG. 5 is a schematic flow chart of method 500 according to anembodiment that may be used to obtain calibration data. The method 500comprises a set 510 in which a plurality of partially overlapping imagesof an object is received from a corresponding plurality of cameras beingarranged along a first and a second direction according to a camerapattern. In step 520, a reference point is selected in an overlap areaof one of the images, and displacement information along the first andthe second direction of the reference point is determined in each of theimages such that a misalignment of the plurality of images along thefirst and the second direction is compensated for by the displacementinformation so as to obtain aligned images. In step 530, offsetinformation between principal points of the plurality of cameras isdetermined, using at least three aligned images. In step 540,calibration data is provided based on the displacement information andbased on the offset information. The calibration data allows forcalibrating the plurality of images so as to comply with the camerapattern when being applied to the images.

FIG. 6a is a schematic flow chart of another method for obtainingcalibration data according to a further embodiment. The method may beorganized in three sub-methods 602, 603 and 608. Block 602 may beconsidered to be an initialization. The initialization 602 may comprisesteps 612, 614 and 616, wherein in step 612 L images are obtained, forexample, from a L=M×N camera array with L=M×N cameras. In anotherexample the L images are obtained from a sphere or cylinder likestructure with L cameras. Two or more of the L cameras may besubstituted by a single camera that is moved between different positionsand configured to provide more than one picture. For simplicity, themethod 600 is first described for the special case of an L=M×N cameraarray with cameras mounted on a rectangular grid.

Sub-method 603 comprises block 604 that is a linear optimizationprocedure and block block 606 that is performed alternatively or inaddition to block 604 and is a non-liner optimization procedure. Onlyone of the blocks 604 and 606 needs to be executed in order to obtain aset of calibration data 607. Block 608 generates, in a block 612, a setof calibrated images from the calibration data obtained in 607 so as toobtain L rectified images in a block 614 or stores the calibration datain a block 616 for future use. Method 600 allows for use of linearstrategies as well as of non-linear strategies and is consistent forplanar camera array and non-planar camera array.

In the following reference will be made to a use of a linearoptimization, for example, in block 604 using steps 612,614 and 616 ofmethod 600. In the M×N camera array (M−1)×N+(M−1)×N, individual stereopairs can be formed using directly adjacent camera positions. Whenreferring, for example, to FIG. 4b , such pairs may be C₁/C₂, C₁/C₄,C₂/C₃, C₂/C₅, C₃/C₆, C₄/C₅, C₄/C₇, C₅/C₆, C₅/C₈, C₆/C₉, C₇/C₈, andC₈/C₉. I.e., the analyzing unit may be configured to determine thedisplacement information pair-wise for a pair of the plurality ofimages, e.g., direct neighbors. Alternatively, any other pairs of imagesmay be used such as C₁/C₅, C₁/C₉, C₂/C₆, or the like. The apparatus isconfigured to select a sufficient number of pairs that allowsdetermining a needed number of parameters. Such a number of parametersmay be, by non-limiting example only 6 as described above. Such a numberis needed for each camera such as 9 such that in total 54 parameters areneeded. Each camera is advantageously part of a number of differentpairs of images that corresponds to the number of dimensions along whichthe camera pattern is arranged. A single line is considered asone-dimensional, camera positions (centers) differing in two directionsare considered as two-dimensional and camera positions (centers)differing in three directions are considered as three-dimensional. Forthe present example, this is true for each camera being part of at leasttwo pairs which is fulfilled for the list given above, wherein a numberof 8 pairs (each of the 9 cameras may for a pair with 8 other camerasand has to be a part of two pairs equals in 2×8/2), wherein additionalinformation may be obtained by additional pairs.

Although description was made in connection with a number of sixparameters that are to be determined in order to calibrate images, alsoa different number of parameters or different parameters may be used,e.g., in case one or more of the above mentioned is considered to bewithin a tolerance range, calibration of that parameter may be skipped.Alternatively or in addition other parameters may be considered such asa distortion of lenses used in the cameras which may lead to low picturequality. The apparatus 10 may be configured to calibrate for thedistortion, e.g., during determination of the displacement information.Different reference points in the image may allow for compensating thedistortion of the lens.

In this way, each view is at least involved in two individual stereopairs. The set of stereo pairs is further referenced as the set Ω. Theset Ω may in example comprise pairs (p,q), (p,r), (q,r) and others. Ingeneral, the method is not limited to directly adjacent cameras. Eachpossible pair of views with overlapping image areas that allow for acommon reference point may be formed. In step 614 interest points orreference are detected on each of the images and these interest pointsare matched on the mentioned stereo pairs in step 616. This may be doneusing any arbitrary feature detection and matching algorithm like SIFT,SURF or BRIEF. As a result, sets of corresponding, matched imagecoordinates

$m_{p,q} = \left\{ {\underset{\underset{m_{p,q}^{(p)}}{}}{u_{p},v_{p}},\underset{\underset{m_{p,q}^{(q)}}{}}{u_{q},v_{q}}} \right\}$

between two images p and q is obtained.

u_(p) and v_(p) denote vectors containing l horizontal/verticalpositions of feature matches between view p and view q respectively. Inthe example of FIG. 3b , l would have dimension 1. If only one set offeature points belonging to a set of cameras is needed, this is denotedby an additional superscript in round braces. I.e., m_(p,q) ^((p))denotes all feature points in image p obtained from the pair (p,q). Thethird (homogenous) component is neglected here but is considered infurther steps. In such a case, the third homogenous component may be setto a value of 1.

In block 604 disparities along a direction, e.g., vertical disparities,for all stereo pairs contained in Ω are reduced or minimized. Forexample, this may be done along the vertical direction such as the ydirection. Formally, for one pair (p,q) this can be written as:

0=v(H(x _(p) ,C _(p) ,C _(q))·m _(p,q) ^((p)))−v(H(x| _(q) ,C _(p) ,C_(q))·m _(p,q) ^((q)))   Formula 14

As before, H(x_(p),C_(p),C_(q)) denotes a function returning a 3×3homography matrix for view p. The argument x_(p) of this function is avector of length k. Function v(x) extracts the vertical component of afeature point coordinate:

${v(x)} = {\frac{x\left( {2,i} \right)}{x\left( {3,i} \right)}{\forall{i \in l}}}$

where x is a 3×l matrix. Consequentially, v(x) returns a vector oflength l. In the present consideration x is a variable unrelated toother variables named x herein and shall simply denote an argument offunction v. The overall minimization task can now be formulated as:

                                  Formulas  15&  16$\underset{x_{1},{x_{2}x_{3}},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{{{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}}}}$     with  x = ∑_(l)x_(l)²

The result thereof may allow the analyzing unit for obtaining a solutionfor all images p, q, r and s forming pairs, the pairs being comprised ofthe set Ω, i.e., a common solution for all x may be obtained.

In Formula 15 the square norm is selected in order to model the errorbetween two views. Other choices are possible. The notationΣ_((p,hd q)∈Ω)∥arg∥ indicates that the tuple (p,q) is one tupleexemplarily selected from the set Ω but also that the summation isperformed for all tuples contained in Ω. H(x_(p),C_(p),C_(q)) is definedas a product of three functions R₀, K and R_(i):

H(x _(p) ,C _(p) ,C _(q))=R ₀ ·K·R _(i) =R ₀ ·{tilde over (H)}(x _(p))

with {tilde over (H)}(x _(p))=K·R _(i)   Formula 17

This formulation can be interpreted as follows: The right-most rotationmatrix R_(i) corrects a camera's orientation such that it corresponds toits ideal orientation as given by the ideal camera setup. The centermatrix K corrects parameters like focal length and principal point. Theleft-most rotation R₀ rotates a camera p such that it forms a stereosystem with a camera q. Only this matrix is individual for each pair ofviews.

K may model the standard camera matrix as given above. R₀ and R_(i)denote the outer and inner rotation matrix. More precisely, R₀ iscomputed from the relative position and orientation of two cameras p andq. Because each camera position is known, R₀ is known. In somearrangements, all these cameras share one common plane, such that theouter rotation R₀ can be modeled using a single rotation around theoptical axis. This may be referred to as angle φ that is explained inconnection with FIG. 3c . As illustrated in FIG. 3a , a 2×2 camera arrayin a general state may comprise camera 18 b that is slightly rotated byangle γ₂, wherein camera 18 c has a slight deviation in focal length.For the pair C₁, C₃, R_(0,13) needs to rotate C₁ and C₃ by −π/2 to forma rectified system assuming that no further distortion occurs. The angleφ may be computed with respect to left/right viewpoint. In the exampleof FIG. 3a , C₁ is exactly above C₃. In order to be rectified to camera18 c and being the left view in this setup, C₁ may be rotated by −90°.In contrast, C₃ forms the right view and is consequently also rotated by−90°. This is also expressed in the argument ordering of H. The secondargument denotes the left view while the third argument denotes theright view.

Alternatively, Formula 17 may also be defined as:

H(x _(p) ,C _(p) ,C _(q))=R ₀ ·R _(i) ·K=R ₀ ·{tilde over (H)}(x _(p))

with {tilde over (H)}(x _(p))=R _(i) ·K   Formula 17b

In the more general case when the cameras do not share a common plane,the apparatus, in particular the analyzing unit is configured todetermine R₀ from the position C_(p) and C_(q) of two cameras p and q.R₀ is a 3×3 matrix with elements as listed below:

Formula 18 R₀(1,1) c_(p) ₁ − c_(q) ₁ R₀(1,2) (c_(p) ₂ − c_(q) ₂ ){square root over (−(c_(p) ₁  − c_(q) ₁  + 1) (−1 + c_(p) ₁  − c_(q) ₁))} R₀(1,3) (c_(p) ₃ − c_(q) ₃ ) {square root over (−(c_(p) ₁  − c_(q) ₁ + 1) (−1 + c_(p) ₁  − c_(q) ₁ ))} R₀(2,1) −(c_(p) ₂ − c_(q) ₂ ) {squareroot over (−(c_(p) ₁  − c_(q) ₁  + 1) (−1 + c_(p) ₁  − c_(q) ₁ ))}R₀(2,2) (1 − c_(p) ₁ + c_(q) ₁ ) c_(p) ₃ ² + 2c_(q) ₃ (−1 + c_(p) ₁ −c_(q) ₁ ) c_(p) ₃ + (1 − c_(p) ₁ + c_(q) ₁ ) c_(q) ₃ ² + c_(p) ₁ − c_(q)₁ R₀(2,3) (c_(p) ₃ − c_(q) ₃ ) (c_(p) ₂ − c_(q) ₂ ) (−1 + c_(p) ₁ −c_(q) ₁ ) R₀(3,1) −(c_(p) ₃ − c_(q) ₃ ) {square root over (−(c_(p) ₁ − c_(q) ₁  + 1) (−1 + c_(p) ₁  − c_(q) ₁ ))} R₀(3,2) (c_(p) ₃ − c_(q) ₃) (c_(p) ₂ − c_(q) ₂ ) (−1 + c_(p) ₁ − c_(q) ₁ ) R₀(3,3) (1 − c_(p) ₁ +c_(q) ₁ ) c_(p) ₂ ² + 2c_(q) ₂ (−1 + c_(p) ₁ − c_(q) ₁ ) c_(p) ₂ + (1 −c_(p) ₁ + c_(q) ₁ ) c_(q) ₂ ² + c_(p) ₁ − c_(q) ₁

The parameter vector x_(p) contains all unknown parameters for a camerap and is defined according to Formula 8. Based on this definitionH(x_(p),C_(p),C_(q)) may now be completely defined as:

                             Formulas  19&  20H(x_(p), C_(p), C_(q)) = R_(o)(0, 0, ϕ_(p, q)) ⋅ K(a_(f), τ_(x), τ_(y)) ⋅ R_(i)(α, β, γ)${{with}\mspace{14mu} \phi_{p}},_{q}{= {a\; \tan \; \frac{C_{p,2} - C_{q,2}}{C_{p,1} - C_{q,1}}}}$

The integer number subscript such as in C_(p,2) denotes the secondelement in the vector C_(p). Please note that in Formula 19 subscriptsin the elements of x_(p) such as α have been neglected. It shouldhowever be clear that in any formula comprising elements of x, thoseelements need to be interpreted with respect to specific camera p. InFormula 19 the ideal orientation of all cameras is identical andperpendicular to the baseline. If this is not the case, H can beextended by an additional rotation matrix R(ρ,α, λ) describing thespecific orientation with respect to the coordinate frame:

H(x _(p) ,C _(p) ,C _(q),ρ,σ,λ)

R _(o)(0,0,φ_(p,q))·R(ρ,σ,λ)⁻¹ ·K(α_(f) p _(x) ,p _(y))·R _(i)(α,β,γ)  Formula 21

In the ongoing description the case expressed in Formula 14 isconsidered. If needed, calculations can be redone including R(ρ,σ, λ).

With respect to the alternative definition of Formula 17 as given inFormula 17b, Formulae 19, 20 and 21 can be adapted by switching theorder of the two rightmost matrices R_(i) and K.

According to embodiments, the apparatus is configured to consider thelens-distortion. The functionality of the apparatus may be expressed byincluding Formula 10 in Formula 11 and Formula 12 as to obtain anundistorted version of reference points m.

The term of Formula 14 forms a non-linear system. This formula can besolved using iterative least-square minimization techniques likeLevenberg-Marquardt algorithm or comparable methods. However, thissolution is inefficient. This non-linear minimization is depicted inblock 606 of FIG. 6 a.

The problem in Formula 12 may be reformulated in order to find thevalues in H efficiently: Therefore, it may be assumed that all variablesin x are small. This corresponds to the block 604 in FIG. 6a . Thisblock is depicted in more detail in FIG. 6b that illustrates alinearization strategy that may be implemented by the analyzing unit.

With the complete definition of H, Formula 12 may be evaluated for onepair of cameras p, q and for one pair of point correspondences.

In the next step such a formulation may be expanded, for example, usingfirst order Taylor expansion of several variables. This expansion may bedefined as:

ƒ(x,a):=Tƒ(x,a)=T F _(x,a)(1,0)=F _(x,a)(0)+F _(x,a)′(0)

F _(x,a) :

→

:=t→ƒ(a+t·(x−a))   Formula 22 & 23

In Formulae 21 and 22 F_(x,a) is a function of t that evaluates thefunction f(x) at the point a+t·(x−a). f(x) is the function toapproximate at some point a. Parameters to the function f(x) arecontained in the parameter vector x. t is a scalar value. a is a vectorwith identical size as x. First order Taylor approximation T of afunction f(x) at some point a may be expressed as Taylor expansion of ascalar function F_(x,a)(t). F_(x,a) is expanded at point t₀=0 bycomputing the function value of F_(x,a) with t₀=0 and the value of thefirst order derivative

$F_{x,a}^{\prime} = {F_{x,a}\frac{d}{dt}}$

also at t₀=0. Evaluating the resulting term at t=1 yields the firstorder Taylor approximation of function f(x) at point a. The expansionmay exemplary be computed for a single pair of images p and q and asingle pair of reference points. Then, Formula 16 (here also includinglens-distortion as given in Formula 9) may be simplified as written inFormula 24.

v(H(x _(p) ,C _(p) ,C _(q))·undist([u _(p) ,v _(p),1]^(T) x _(p)))−v(H(x_(q) ,C _(p) ,C _(q))·undist([u _(g) ,v _(q),1]^(T) ,x _(q)))   Formula24

The apparatus may be configured to determine the Taylor expansion usingFormula 22 for Formula 24 for all variables contained in x_(p) and x_(q)forming a parameter vector χ like χ=[x_(p), x_(q)]. χ is argument toFormula 22 as well as an expansion point a=0. Without limitation, otherexpansion points may be selected.

This yields an approximation for the point x_(p)=0 and x_(q)=0. Theresulting equation can be differentiated with respect to every unknownvariable in x_(p) and x_(q). As a result, one obtains the Jacobianmatrix of this system. Each column in the Jacobian matrix contains anapproximation for each parameter in the system. These approximations mayalso be expressed as:

Formula 25 α_(p) −v_(p)u_(p) sin(φ_(pq)) − cos(φ_(pq)) v_(p) ² −cos(φ_(pq)) α_(q) v_(q)u_(q) sin(φ_(pq)) + cos(φ_(pq)) v_(q) ² +cos(φ_(pq)) β_(p) sin(φ_(pq)) u_(p) ² + v_(p)u_(p) cos(φ_(pq)) +sin(φ_(pq)) β_(q) −sin(φ_(pq)) u_(q) ² − v_(q)u_(q) cos(φ_(pq)) −sin(φ_(pq)) γ_(p) u_(p) cos(φ_(pq)) − v_(p) sin(φ_(pq)) γ_(q) −u_(q)cos(φ_(pq)) + v_(q) sin(φ_(pq)) a_(f) _(p) sin(φ_(pq)) u_(p) +cos(φ_(pq)) v_(p) a_(f) _(q) −sin(φ_(pq)) u_(q) − cos(φ_(pq)) v_(q){open oversize brace} τ_(x) _(p) sin(φ_(pq)) {close oversize brace}{open oversize brace} τ_(x) _(q) −sin(φ_(pq)) {close oversize brace}τ_(y) _(p) cos(φ_(pq)) τ_(y) _(q) −cos(φ_(pq)) k_(p,1) (sin(φ_(pq))u_(p) + cos(φ_(pq)) v_(p)) {square root over (u_(p) ² + v_(p) ²)}k_(q,1) −(sin(φ_(pq)) u_(q) + cos(φ_(pq)) v_(q)) {square root over(u_(q) ² + v_(q) ²)} k_(p,2) (u_(p) ² + v_(p) ²) (sin(φ_(pq))u_(p + cos(φ) _(pq)) v_(p)) k_(q,2) −(u_(q) ² + v_(q) ²) (sin(φ_(pq))u_(q + cos(φ) _(pq)) v_(q))

In this case, k_(g, h) denotes the parameters to model lens-distortionof view g being either q or p according to Formula 10 and 11 and thedefinition of vector κ.

The remaining constant part may be formulated as:

b _(p,q)=(v _(p) −v _(q))cos(φ_(pq)))+sin(φ_(pq))(u _(p) −u _(q))  Formulas 26

All elements in J and also the constant part b_(p,q) can now beevaluated for point correspondences. The resulting system can bere-written as linear system according to:

$\begin{matrix}{{J_{p,q} \cdot \begin{bmatrix}{\overset{\sim}{x}}_{p}^{t} \\{\overset{\sim}{x}}_{q}^{T}\end{bmatrix}} = {{{- {b_{p,q}\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack}} \cdot \begin{bmatrix}{\overset{\sim}{x}}_{p}^{t} \\{\overset{\sim}{x}}_{q}^{T}\end{bmatrix}} = {- b_{p,q}}}} & {{{{Formulas}\mspace{14mu} 27}\&}\mspace{14mu} 28}\end{matrix}$

In Formulas 27 & 28, J_(p,q) is a lx(2*m) matrix where l denotes thenumber of point correspondences and m denoting the number of variablesin x. b is a column vector with length l.

As the Taylor expansion only approximates the function value at onepoint, the true parameter vector x_(p) needs to be replaced by anapproximated value {tilde over (x)}_(p). In Formula 27, J_(p,q) andb_(p,q) denote the Jacobian matrix and constant part of two views (p, q)evaluated for all point correspondences. As before, J_(p,q) ^((p)) inFormula 28 denotes the columns of J_(p,q) belonging to unknowns ofcamera p.

For the set of all stereo-pairs Ω, the overall minimization task may beformulated by the analyzing unit 24 as follows: Each pair (p, q) in Ωforms one line in the global constraint matrix A that is generated in astep 621 of method 600. For a 2×2 camera array with four individualstereo pairs, the problem can be written as:

$\begin{matrix}{{{\underset{\underset{A}{}}{\begin{bmatrix}J_{12}^{(1)} & J_{12}^{(2)} & 0 & 0 \\J_{13}^{(1)} & 0 & J_{13}^{(3)} & 0 \\0 & J_{24}^{(2)} & 0 & J_{24}^{(4)} \\0 & 0 & J_{34}^{(3)} & J_{34}^{(4)}\end{bmatrix}} \cdot \underset{\underset{\Delta}{}}{\begin{bmatrix}{\overset{\sim}{x}}_{1}^{T} \\{\overset{\sim}{x}}_{2}^{T} \\{\overset{\sim}{x}}_{3}^{T} \\{\overset{\sim}{x}}_{4}^{T}\end{bmatrix}}} = \underset{\underset{b}{}}{- \begin{bmatrix}b_{12} \\b_{13} \\b_{24} \\b_{34}\end{bmatrix}}}{{A \cdot \Delta} = b}} & {{{{Formulas}\mspace{14mu} 29}\&}\mspace{14mu} 30}\end{matrix}$

The number of feature matches in each stereo pair may be greater thanthe number of unknown variables in x, such that the overall system isoverdetermined. To solve the linearized system and thus finding a set ofparameters approximating the real condition of the plurality of camerasas given by Formula 15, the system can be solved for Δ by computing thePseudo-inverse of A. In total, the solution for Δ can be obtained as:

Δ=−(A ^(T) ·A)⁻¹ ·A ^(T) ·b   Formula 31

Thus, the analyzing unit may be configured to minimize the error basedon the determination rule that corresponds to Formula 31. The analyzingunit may be configured for determining the displacement informationusing a set of parameters, that indicates a real condition of theplurality of cameras. The parameters may comprise a non-linearrelationship with respect to each other. The analyzing unit may beconfigured to determine a linearized version of the parameters in thevector Δ and may be configured to determine the displacement informationby minimizing an error of the linearized version Δ with respect to adesired condition of the plurality of cameras. This is indicated as step622, i.e., the system is solved for Δ. Alternatively, the analyzing unitmay solve the non-linear problem. The at least two elements in x showthe approximation for τ_(x) and τ_(y). As both do not depend on imagecoordinates and only depend on φ which is constant, the linearizationdoes not provide valuable information of these parameters. For thisreason, both parameters τ_(x) and τ_(y) are excluded in this step. Thisworks by dropping the value in {tilde over (x)}_(p) and {tilde over(x)}_(q) as well as the appropriate columns in J_(p,q). As aconsequence, deviations in a camera's principal point will beapproximated by a small additional rotation around the y axis and x axisrespectively. I.e., the calibration data may be based on angles thatdescribe a rotation of the cameras, to a focal length of the camera andto a principle point of the cameras, wherein the calibration data may bein absence to a position of the cameras of the plurality of cameras,i.e., does possibly not contain information indicating the camerapositions. This may be possible as the camera position is considered asknown variable in the system.

The elements in Δ now contain an approximated solution for the overallnon-linear problem. At this point, the overall problem can now be solvedusing an iterative optimization algorithm like gradient descent. Inevery iteration a small portion of ε·Δ is added to the previous x_(p)^((t-1)) forming a new x_(p) ^((t)), i.e., the solution vector isupdated in a step 623. The current x_(p) ^((t)) is then used to generatean intermediate mapping homography {tilde over (H)} for each camera.This intermediate homography is used to update all feature sets (p,q)∈Ω. Those updated feature sets {tilde over (m)}_(p,q) form new initialfeature sets in the next iteration, i.e., the feature points aretransformed in a step 624. Ideally, x_(p) ^((t)) iteratively approachesthe desired solution. As soon as Δ is smaller than threshold {tilde over(ε)}₀ or the remaining error cannot be reduced anymore, the algorithmstops. I.e., the analyzing unit may be configured to iterativelyminimize the error of the linearized version. This is indicated in step625 checking, if congruence is reached. This may also be expressed as:

x _(p) ^((t)) =ε·Δ+x _(p) ^((t-1))

{tilde over (m)} _(p,q) ^((p)) ={tilde over (H)} _(p) ·m _(p,q) ^((p))=K(a _(f,p) ,p _(x,p) ,p _(y,p))·R _(i)(α_(p),β_(p),γ_(p))·m _(p,q)^((p))

and {tilde over (m)} _(p,q) ^((q)) ={tilde over (H)} _(p) ·m _(p,q)^((q)) =K(a _(f,q) ,q _(x,q) ,q _(y,q))·R _(i)(α_(q),β_(q),γ_(q))·m_(p,q) ^((q))   Formulas 32 & 33(a)

Once this iterative minimization algorithm has converged, the secondpart of this linearized calibration procedure can be started, i.e.,block 627 may be started, e.g., by the determining unit. At the end ofblock 625 a result according to FIG. 3c may be obtained, i.e., after thefirst rectification step, images may be pair-wise rectified.Corresponding pixels are shifted horizontally only for views in the samerow and shifted vertically in the same column respectively. Both, thetransformation of the images in step 622 by the analyzing unit and theestimation of the principal points in step 627 by the determining unitmay be executed by determining a solution for a minimization problemthat may be linear or non-linear. I.E., four problems may be formulated,two for solving the minimization criteria with respect to thetransformation that may be referred to a first (vertical) disparity andtwo for solving the second (horizontal) disparity. The problems may bere-formulated for a 2D and a 3D-pattern of cameras.

The final state after the converge criteria is this illustrated FIG. 3cfor a 2×2 array. The position of the marked objects in C₁ is denoted byu₁ and v₁. In C₂, the object has slightly moved to the left but thevertical coordinate is still v₁. A similar condition holds for thevertical pair C₁, C₃: the object is shifted vertically but nothorizontally.

As formally expressed in Formula 13, corresponding disparity values inthree cameras may depend on the array geometry only. Up to this point,this has not been taken into account in the method 604. Consequentially,it is possible that Formula 13 is not met. Simplified, a distancebetween columns in pairs may differ or vary. This condition can bere-written in order to formulate a condition similar to Formula 15. Fora triplet of three cameras this may be formulated as:

                                   Formula  34&  35${{\left( {{u\left( {{R_{o}\left( {0,0,\phi ,_{p,q}} \right)} \cdot {\overset{\sim}{m}}_{q}} \right)} - {u\left( {{R_{o}\left( {0,0,\phi ,_{p,q}} \right)} \cdot {\overset{\sim}{m}}_{q}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\left( {{u\left( {{R_{o}\left( {0,0,\phi ,_{p,q}} \right)} \cdot {\overset{\sim}{m}}_{p}} \right)} - {u\left( {{R_{o}\left( {0,0,\phi ,_{p,r}} \right)} \cdot {\overset{\sim}{m}}_{r}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}} = 0$     with $\mspace{79mu} {\overset{\sim}{m} = {\begin{bmatrix}\overset{\sim}{u} & \overset{\sim}{v} & 1\end{bmatrix}^{T} = {{\overset{\sim}{H}(x)} \cdot m}}}$

H denotes the mapping homography that has been obtained in the lastiteration of the previous steps, i.e., block 625. H(⋅) may be defined asgiven in Formula 17 or 17b with respect to a parameter vector x. Thefunction u(⋅) extracts the horizontal component of image point.d(C_(p)−C_(q)) denotes the Euclidian distance between two camera C_(p)and C_(q). Using {tilde over (m)} instead of m, it may be assumed thatoptical axes are parallel and that all cameras have identical focallengths. The remaining error can be modeled by a shift in the cameras'principal point. Formula 34 can now be used to define a secondminimization problem which is subject to be minimized and formulated asfollows:

${{\mspace{664mu} {{{Formula}\mspace{14mu} 36}{\underset{x_{1},x_{2},{x_{3}{\ldots x}_{N}}}{\arg \; \min}\sum\limits_{{({p,q,r})} \in \Psi}}}\quad}{{{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} -}\quad}\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{r},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(r)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}$

Formula 36 refers to the general case when the cameras are not in acalibrated state. The analyzing unit may be configured to solve thisproblem so as to determine the displacement information. Alternativelyor in addition, the analyzing unit may be configured to solve alinearized version thereof that may be expressed by

${\mspace{664mu} {{{Formula}\mspace{14mu} 37}{\underset{x_{1},x_{2},{x_{3}{\ldots x}_{N}}}{\arg \; \min}\sum\limits_{{({p,q,r})} \in \Psi}}}\quad}{\quad{{{\left( {{u\left( {{\hat{H}\left( {{\hat{x}}_{p},C_{p},C_{q}} \right)} \cdot {\overset{\sim}{m}}_{p,q,r}^{(p)}} \right)} - {u\left( {{\overset{\sim}{H}\left( {{\hat{x}}_{q},C_{p},C_{q}} \right)} \cdot {\overset{\sim}{m}}_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\quad{\left( {{u\left( {{\hat{H}\left( {{\hat{x}}_{p},C_{p},C_{r}} \right)} \cdot {\overset{\sim}{m}}_{p,q,r}^{(p)}} \right)} - {{\quad\quad}{u\left( {{\hat{H}\left( {{\hat{x}}_{r},C_{p},C_{r}} \right)} \cdot {\overset{\sim}{m}}_{p,q,r}^{(r)}} \right)}}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}}}}}$

Function Ĥ(x_(p),C_(p),C_(q)) may be defined as:

{tilde over (H)}({circumflex over (x)} _(p) ,C _(p) ,C _(q))

R _(o)(0,0,φ_(p,q))·K _(i)(0,{circumflex over (τ)}_(x,p),{circumflexover (τ)}_(y,p))

{circumflex over (x)}

[{circumflex over (τ)}_(x),{circumflex over (τ)}_(y)]   Formula 38

Function Ĥ as given in Formula 38 models a transformation that performsa shift in a cameras' principal point τ and as before rotates two imagessuch that they may form a stereo pair. Thus, by comparing two individualstereo pairs minimizing the remaining error as given in Formulas 36 and37 by finding optimum values for {circumflex over (τ)} may be performed.

As an additional condition, Formulas 36 and 37 may be minimized withrespect to the minimization criteria given in Formula 15. Beingconsistent to the definition of sets of feature point pairs m_(p,q)given before, m_(p,q,r) defines a set of triple matching feature points.All triplets (p, q, r) of cameras sharing overlapping image areas arecontained in the set ψ.

Formulas 36 and 37 may be evaluated for various camera configurationswith the requirements as given above. Depending on the cameraconfiguration and the resulting image overlap the set of equations thatis obtained will have a different shape. It is possible that theresulting system can be solved exactly with linear methods, and is overor under determined.

In matrix notation, problem 37 may be written as follows: For onetriplet of cameras (p,q,r) this may be formulated as:

$\begin{matrix}{\mspace{664mu} {{{{{Formula}\mspace{14mu} {{39\mspace{79mu}\begin{bmatrix}A_{pqr}^{(p)} & A_{pqr}^{(q)} & A_{pqr}^{(r)}\end{bmatrix}} \cdot \begin{bmatrix}{\hat{x}}_{p} & {\hat{x}}_{q} & {\hat{x}}_{r}\end{bmatrix}^{T}}} + b_{pqr}} = 0}\mspace{79mu} {{A \cdot \begin{bmatrix}{\hat{x}}_{p} & {\hat{x}}_{q} & {\hat{x}}_{r}\end{bmatrix}^{T}} + b_{pqr}}\mspace{664mu} {{Formula}\mspace{14mu} 40}}} \\{\mspace{79mu} {wherein}} \\{{{A_{pqr}^{(p)} = \begin{bmatrix}\frac{{{- {\cos \left( \phi_{pr} \right)}} \cdot d_{pq}} + {{\cos \left( \phi_{pq} \right)} \cdot d_{pr}}}{d_{pq} \cdot d_{pr}} & \frac{{{\sin \left( \phi_{pr} \right)} \cdot d_{pq}} - {{\sin \left( \phi_{pq} \right)} \cdot d_{pr}}}{d_{pq} \cdot d_{pr}} \\{\sin \left( \phi_{pq} \right)} & {\cos \left( \phi_{pq} \right)} \\{\sin \left( \phi_{pr} \right)} & {\sin \left( \phi_{pr} \right)} \\0 & 0\end{bmatrix}}\mspace{664mu} {{Formula}\mspace{14mu} 41}\mspace{79mu} {A_{pqr}^{(q)} = {{\begin{bmatrix}\begin{matrix}\frac{\cos \left( \phi_{pq} \right)}{d_{pq}} \\{- {\sin \left( \phi_{pq} \right)}} \\0 \\{\sin \left( \phi_{qr} \right)}\end{matrix} & \begin{matrix}\frac{\sin \left( \phi_{pq} \right)}{d_{pq}} \\{- {\cos \left( \phi_{pq} \right)}} \\0 \\{\cos \left( \phi_{qr} \right)}\end{matrix}\end{bmatrix}\mspace{31mu} A_{pqr}^{(r)}} = \begin{bmatrix}\begin{matrix}\frac{\cos \left( \phi_{pr} \right)}{d_{pr}} \\0 \\{- {\sin \left( \phi_{pr} \right)}} \\{- {\sin \left( \phi_{qr} \right)}}\end{matrix} & \begin{matrix}\frac{\sin \left( \phi_{pq} \right)}{d_{pq}} \\0 \\{- {\cos \left( \phi_{pr} \right)}} \\{- {\cos \left( \phi_{qr} \right)}}\end{matrix}\end{bmatrix}}}}\mspace{79mu} {b_{pqr} = \begin{bmatrix}{\frac{{- {\overset{\sim}{u}}_{p}} + \overset{\sim}{u_{q}}}{d_{pq}} + \frac{{\overset{\sim}{v}}_{p} - {\overset{\sim}{v}}_{r}}{d_{pr}}} \\0 \\0 \\0\end{bmatrix}}}\end{matrix}$

In matrix A the first line arises from Formula 36 evaluated for atriplet (p,q,r). Lines 2 to 4 may arise from the additional requirementas given by Formula 15. As transformed points {tilde over (m)} are usedhere, Formula 15 simplifies and only R_(o) (or the elements comprised inR_(o)) remain.

For a camera pattern comprising more than 3 cameras, and so more thanone camera-triplet in the set ψ, each triplet forms one line in a globalmatrix A_(G).

                                 Formula  42${A_{G} = \begin{bmatrix}A_{pqr}^{(p)} & A_{pqr}^{(q)} & A_{pqr}^{(r)} & 0 & 0 & \ldots & 0 \\A_{prs}^{(p)} & 0 & A_{prs}^{(r)} & A_{prs}^{(s)} & 0 & \ldots & 0 \\0 & A_{qrs}^{(p)} & A_{qrs}^{(r)} & A_{qrs}^{(s)} & 0 & \ldots & 0 \\A_{prt}^{(p)} & 0 & A_{prt}^{(r)} & 0 & A_{prt}^{(t)} & \ldots & 0 \\\ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots\end{bmatrix}};\; {b_{G} = \begin{bmatrix}b_{pqr} \\b_{prs} \\b_{qrs} \\b_{prt} \\\ldots\end{bmatrix}}$

In Formula 42, the set ψ comprises four triplets denoted as pqr, prs,qrs and prt. A solution for the equation system may then be found as:

[{circumflex over (x)} _(p) {circumflex over (x)} _(q) {circumflex over(x)} _(r) . . . ]^(T)=−(A _(G) ^(T) ·A _(G))⁻¹ ·A ^(T) _(G) ·b _(G)  Formula 43

A critical setup needs to be discussed in the case when only threecameras or images are available. The presented strategy can easily beextended to such a scenario. Evaluating the sum of Formula 36 for thecamera triplet (C₁, C₂, C₃) as given in FIG. 3a yields:

$\begin{matrix}{{\frac{u_{1} + {\hat{\tau}}_{x,1} - u_{2} - {\hat{\tau}}_{x,2}}{d\left( {C_{1},C_{2}} \right)} - \frac{{- v_{1}} - {\hat{\tau}}_{y,1} - v_{3} - {\hat{\tau}}_{y,3}}{d\left( {C_{1,}C_{3}} \right)}} = 0} & {{Formula}\mspace{14mu} 44}\end{matrix}$

Evaluating Formula 15 for the pairs (C₁, C₂), (C₁, C₃) and (C₂, C₃)yields three additional conditions:

v _(i)+{circumflex over (τ)}_(y,1) −v ₂−{circumflex over (τ)}_(y,2)=0

−u ₁−{circumflex over (τ)}_(x,1) +u ₃+{circumflex over (τ)}_(x,3)=0

−(u ₂ +v ₂+{circumflex over (τ)}_(x,2)+{circumflex over (τ)}_(y,2) −u ₃−v ₃ −{circumflex over (τ)}t _(x,3)−{circumflex over (τ)}_(y,3)=0  Formula 45

Rewritten in matric notation, Formula 39 becomes:

$\begin{matrix}{{{\underset{A}{\underset{}{\begin{bmatrix}\begin{matrix}\frac{1}{d\left( {C_{1},C_{2}} \right)} \\0 \\{- 1} \\0\end{matrix} & \begin{matrix}\frac{1}{d\left( {C_{1},C_{3}} \right)} \\1 \\0 \\0\end{matrix} & \begin{matrix}\frac{1}{d\left( {C_{1},C_{2}} \right)} & 0 & 0 \\0 & {- 1} & 0 \\0 & 0 & 1 \\{- 1} & {- 1} & 1\end{matrix} & \begin{matrix}\frac{1}{d\left( {C_{1},C_{3}} \right)} \\0 \\0 \\1\end{matrix}\end{bmatrix}}} \cdot}\quad}{\quad{\begin{bmatrix}{\hat{\tau}}_{x,1} \\{\hat{\tau}}_{y,1} \\{\hat{\tau}}_{x,2} \\{\hat{\tau}}_{y,2} \\{\hat{\tau}}_{x,3} \\{\hat{\tau}}_{y,3}\end{bmatrix} = \begin{bmatrix}{\frac{u_{2} - u_{1}}{d\left( {C_{1},C_{2}} \right)} + \frac{v_{1} - v_{3}}{d\left( {C_{1},C_{3}} \right)}} \\{v_{2} - v_{1}} \\{u_{1} - u_{3}} \\{u_{2} + v_{2} - u_{3} - v_{3}}\end{bmatrix}}}} & {{Formula}\mspace{14mu} 46}\end{matrix}$

Formulas 44, 45 and 46 may be evaluated for one triplet-pointcorrespondence m_(1,2,3)={u₁,v₁,u₂,v₂,u₃,v₃}. The resulting system hassix unknown variables but only four formulas. It is thereforeunder-determined. Additionally, the matrix A does not depend on thevalues of u and v. This means that by adding additional triplet-pointcorrespondences the matrix A cannot obtain full row rank. Any line addedto A will only be a duplicate of an already existing line. Therefore,this system cannot be solved without further assumptions or withoutreducing the number of unknown variables. One possibility to reduce thenumber of unknowns is to define C₁ as reference camera with {circumflexover (τ)}_(x,1)={circumflex over (τ)}_(y,1)=0. This allows removal ofthe first and second column of matrix A. The remaining linear system canthen easily be solved using linear methods. This is shown by:

                                      Formula  47${\underset{\underset{A}{}}{\begin{bmatrix}\; & \begin{matrix}\frac{- 1}{d\left( {C_{1},C_{2}} \right)} & 0 & 0 \\0 & {- 1} & 0 \\0 & 0 & 1 \\{- 1} & {- 1} & 1\end{matrix} & \begin{matrix}\frac{1}{d\left( {C_{1},C_{3}} \right)} \\0 \\0 \\1\end{matrix}\end{bmatrix}} \cdot \left\lbrack \begin{matrix}{\hat{\tau}}_{x,2} \\{\hat{\tau}}_{y,2} \\{\hat{\tau}}_{x,3} \\{\hat{\tau}}_{y,3}\end{matrix} \right\rbrack} = {\quad\begin{bmatrix}{\frac{u_{2} - u_{1}}{d\left( {C_{1},C_{2}} \right)} + \frac{v_{1} - v_{3}}{d\left( {C_{1},C_{3}} \right)}} \\{v_{2} - v_{1}} \\{u_{1} - u_{3}} \\{u_{2} + v_{2} - u_{3} - v_{3}}\end{bmatrix}}$

This strategy can be extended towards a general planar camera array.Depending on the field of view of each camera, their position andorientation, there may exist cameras that are unable to be formed intotriplet matches over all possible camera combinations. It is, ingeneral, therefore not possible to predict how many triplets and pairscan be formed. The amount of linear independent equations one obtains tosolve for each principal point is unknown. This needs to be taken intoaccount when setting up an array.

Alternatively, the linearization of Formula 36 may also be expressed as:

${\underset{x_{1},x_{2},{x_{3}{\ldots x}_{N}}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{\begin{bmatrix}J_{p,q,r}^{(p)} & J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{bmatrix} \cdot \begin{bmatrix}x_{p} \\\begin{matrix}x_{q} \\x_{r}\end{matrix}\end{bmatrix}}}} + b_{p,q,r}$

Wherein the argument of Formula 36 is approximated as

$\left( {{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\left. \quad {\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{r},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(r)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}} \right) 2 {\quad \approx \quad}{\quad{{\begin{bmatrix}J_{p,q,r}^{(p)} & J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{bmatrix} \cdot \begin{bmatrix}x_{p} \\\begin{matrix}x_{q} \\x_{r}\end{matrix}\end{bmatrix}} + b_{p,q,r}}}}} \right.$

Alternatively, the analyzing unit may be configured to determine thedisplacement information, to minimize the error respectively based on anon-linear concept. This may lead to a slightly increase incomputational effort but may in return allow for usage of commonavailable implementations as in software libraries or the like. In caseof non-linear optimization both problems as given in Formula 15 andFormula 36 are subject to be minimized with respect to vectors x.

Based on {circumflex over (x)}_(p),x_(p),C_(p) and C_(q) and as part ofas 612 and/or 616 the total calibrating function H being referred to asa homography matrix can be computed as:

                                 Formula  48${H\left( {x_{p},{\hat{x}}_{p},C_{p},C_{q}} \right)}\overset{def}{=}{\underset{{view}\mspace{14mu} {dependend}}{\underset{}{R_{o}\left( {0,0,\phi_{p,q}} \right)}} \cdot \underset{constant}{\underset{}{K_{i}{\left( {a_{f},{\hat{\tau}}_{x},{\hat{\tau}}_{y}} \right) \cdot {R_{i}\left( {\alpha,\beta,\gamma} \right)}}}}}$

H(⋅) may thus be a function providing a homography matrix undistortingpoints according to a parameter vector x_(p) and camera positions C_(p)and C_(q).

Formula 48 relates to the definition of H as given in Formula 17.Alternatively, the definition as given in Formula 17b may be used.

One can see that the right part is constant for a camera p while theleft part depends on the relative position of both cameras. Anapplication of the calibration data, i.e., the homography in step 612,allows for obtaining rectified, i.e., calibrated images in a step 614.In the special scenario of rectangular arrays as used as an exampleherein, this can be exploited to increase performance. In a rectangularsystem matrix, R₀ is either the identity matrix or a rotation around90°. It is therefore beneficial to apply the constant part once for animage and execute the second part on demand. Within a computer system, arotation around 90° can easily be executed by swapping rows and columnsof an image without further computations. In other words, FIG. 6 shows aschematic working principle of a calibration method. The left columncovers initialization, the central column corrects focal length andorientation whereas the right column corrects deviations in theprincipal points.

For the following calibration procedure, there may be assumptions thatall cameras in the array are mechanical mounted on a strict, staticgrid. Therefore, it may be assumed that it has a good pre-calibration.This mechanical setup may be, in general, not aligned at pixel level andcomprise slight deviations that remain in the images. As illustrated inFIGS. 2a and 3b , the orientation of the cameras may differ as well astheir focal length. Due to this mechanical pre-calibration, the cameraposition may be considered as known and pairs of cameras may be close tothe rectified state. Therefore, the camera matrix K for a camera may bemodelled as:

$\begin{matrix}{K = \begin{pmatrix}{f \cdot \left( {1 + a_{f}} \right)} & s & \tau_{x} \\0 & {f \cdot \left( {1 + a_{f}} \right)} & \tau_{y} \\0 & 0 & 1\end{pmatrix}} & {{Formula}\mspace{14mu} 49}\end{matrix}$

wherein f denotes a normalized focal length and may be set to 1. atdenotes a small deviation of focal length. Furthermore, the focal lengthmay be considered as constant in horizontal and vertical direction,i.e., s_(i)≡0 which means that square pixels are present and a_(f)<<f=1.As the camera's principal point, one can further assume that τ_(x)≈0 andτ_(y)≈0. The orientation is modeled using a standard rotation matrixR(α,δ,γ) with α, β and γ denoting tilt, yaw and roll respectively. Inthis case, it may be assumed that |α|, |β|, |γ|<<1, wherein all anglesmay be given in radiants). At the end of the calibration, deviationsfrom the ideal model are known.

Details of above embodiments may relate to a 2D camera pattern. Whendetermining calibration data for a 3D camera pattern, the same or atleast related considerations may be taken into account.

In a general case when cameras do not reside on a common plane, i.e.,form a 2D-pattern, Formula 13 cannot be applied and needs to be replacedby a more general formulation that fits to a 3D-pattern. Thisformulation is known as the Trifocal-Tensor and is presented in [2]. Thebasic equation is also expressed in Formula 45. The Formula states thatgiven three corresponding image points m_(p), m_(q) and m_(r) visible inthree cameras the function results in a 3×3 zero matrix. This alsoentails that camera properties like position, orientation and intrinsicsare known precisely.

$\begin{matrix}{{\left\lbrack m_{q} \right\rbrack_{x}{\left( {\underset{1}{\sum\limits^{3}}{m_{p}^{i} \cdot T_{{({p,q,r})},i}}} \right) \cdot \left\lbrack m_{r} \right\rbrack_{x}}} = 0_{3 \times 3}} & {{Formula}\mspace{14mu} 50}\end{matrix}$

Formula 50 may be expanded as:

[m _(q)]_(x)·(u _(p) ·T _((p,q,r),1) +v _(p) ·T _((p,q,r),2) +T_((p,q,r),3))·[m _(r)]_(x)=0_(3×3)   Formula 51

In Formula 45 and Formula 46 [a]_(x) denotes a skew symmetric matrixcorresponding to some vector a as:

$\begin{matrix}{\lbrack a\rbrack_{x}:=\begin{pmatrix}0 & {- a_{3}} & a_{2} \\a_{3} & 0 & {- a_{1}} \\{- a_{2}} & a_{1} & 0\end{pmatrix}} & {{Formula}\mspace{14mu} 52}\end{matrix}$

The three elements T_((p,q,r),1,), T_((p,q,r),2) and T_((p,q,r),3)denote a representation of the Trifocal Tensor. Each elementT_((p,q,r),i) may be represented as a 3×3 matrix and its entries may becomputed (according to [2]) from three projection matrices P_(q), P_(q)and P_(r). Based on the ideal layout and ideal extrinsic and intrinsiccamera parameters it may be assumed that all projection matrices andconsequentially all Tensor elements may be computed.

Due to mechanical misalignment, measured points in images p, q and rwill not obey this ideal setup and consequentially computing Formula 51for some triplet of corresponding points and idealized Tensor elementsmay result in values different from a zero matrix.

As in the planar system Formula 46 may be modified as follows:

[m _(q) ^(#)]_(x)·(u _(p) ^(#) ·T _((p,q,r),1) +v _(p) ^(#) ·T_((p,q,r),2) +T _((p,q,r),3))·[m _(r) ^(#)]_(x)=0_(3×3)   Formula 53

g(x _(p) ,x _(q) ,x _(r)):=[{tilde over (H)}(x _(q))·m _(q)]_(x)·(u _(p)^(#) ·T _((p,q,r),1) +v _(p) ^(#) ·T _((p,q,r),2) +T_((p,q,r),3))·[{tilde over (H)}(x _(r))·m _(r)]_(x)   Formula 54

wherein

m ^(#)=[u ^(#) v ^(#)1]^(T) ={tilde over (H)}(x _(p))·m   Formula 55

A calibrated image point m_(p) ^(#) may be obtained by transforming anuncalibrated image point m_(p) by a mapping homography as m_(p)^(#)={tilde over (H)}(x_(p))·m_(p)·{tilde over (H)}(x_(p)) may be againbe constructed from a camera matrix K and a rotation matrix R as inFormula 17 or 17b with parameters contained in a parameter vector x.

As before, an additional element to compensate for lens-distortion maybe integrated in order to obtain calibrated image points m_(p) ^(#).

m _(p) ^(#)(x _(p)):={tilde over (H)}(x _(p))·undist(m _(p) ,x _(q))  Formula 56

Formula 54 may therefore also be represented as function of x_(p), x_(q)and x_(r):

g(x _(p) ,x _(q) ,x _(r)):=[m _(q) ^(#)]_(x)·(u _(p) ^(#) ·T_((p,q,r),1) +v _(p) ^(#) ·T _((p,q,r)2) +T _((p,q,r)3))·[m _(r)^(#)]_(x)   Formula 57

For a general set of cameras, the apparatus may be configured to solvetwo optimization problems simultaneously, wherein one of theoptimization problems may be subject to be minimized according to:

$\begin{matrix}{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{{g\left( {x_{p},x_{q},x_{r}} \right)}}}} & {{Fromula}\mspace{14mu} 58}\end{matrix}$

In Formula 53, Ψ may denote a set containing all sets of camera tripletsrelevant for analyzation. As before, it is an example target to findvectors x that simultaneously optimize Formula 15 and Formula 53, e.g.,in a linearized case. The problem of formula 53 may be transferred to alinearized version and solved as such a linearized version, such asdefined in formula 56.

Again, the apparatus may be configured to execute a non-linearoptimization method or a linearized optimization method. This isidentical as presented for the planar case as depicted in FIG. 6a-b .The apparatus may be configured to use a non-linear optimization methodso as to minimize the error. In case of a non-linear optimizationmethod, the analyzing unit may be configured to optimize the resultsindicated in Formulae 15 and 58 simultaneously yielding optimum vectorsx.

For the linearization, a local sub-problem consisting of three camerasp,q and r is evaluated for each element of Formula 57. This yields 9functions of x_(p), x_(q) and x_(r). Each of these functions can beapproximated using Taylor expansion as defined in Formulae 22 and 23.Formally, this can be expressed as:

g _(ij)(x _(p) ,x _(q) ,x _(r))≈Tg _(i,j)(χ,a)

χ=[x _(p) ,x _(q) ,x _(r)]   Formula 59 & 60

The approximated elements g_(i,j) may be composed to a vector andsubsequently re-written in matrix notation yielding a Jacobian matrixand a column vector b. As a result a linearized system is obtained withunknowns in χ. As before, a is a zero-vector with the same length as χ.

0=[Tg _(1,1)(χ,a),Tg _(1,2)(χ,a), . . . ,Tg _(3,3)(χ,a)]^(T)

0=J·χ+b   Formula 61 & 62

T g_(i,j)(χ, a) denotes the Taylor approximation of a function g andshould not be mixed up with the Tensor elements in Formula 57. Theindices in g denote an entry in the matrix returned by function g.(Compare also for Formula 53 and 54). However it should be clear theTensors are elements of Formulae 58 to 62 and further on.

Alternatively, the analyzing unit may be configured to linearly minimizethe error. In case of a linearized optimization method as shown in FIG.6b Formula 57 may be linearized using first-order Taylor expansionresulting in a linearized approximation of the overall non-linearproblem for some distinct, approximated point. As before, the obtainedlinearized equations may be represented as an Jacobian matrix J and aconstant part b. This Jacobian matrix is composed from three submatricesas:

J _(p,q,r)=[J _(p,q,r) ^((p)) J _(p,q,r) ^((q)) J _(p,q,r) ^((r))]  Formula 63

In the same fashion as presented in Formula 29, the linearized globalproblem can be constructed from a series of local, linearized problems.Therefore, each camera triplet contained in Ψ may be represented so asto form one line in a matrix A_(T) and a vector b_(T) containing theconstant part. The apparatus may be configured to form the Jacobianmatrix accordingly, but according to other embodiments, it may form amatrix differently. In particular, the apparatus may be configured fordetermining an approximation for solving the first and second basisproblems using the coordinates C of the cameras and of the referencepoints m on the basis of Jacobian matrices. The matrix A_(T) may then beconcatenated with matrix A (further referenced to as A_(B)) as obtainedfrom Formula 29 for all stereo pairs (p,q) contained in the set Ω. Inthe same fashion, vectors b (also referenced to as b_(B)) and b_(T) maybe concatenated.

$\begin{matrix}{A_{G} = {\begin{bmatrix}A \\A_{T}\end{bmatrix} = \begin{bmatrix}A_{B} \\A_{T}\end{bmatrix}}} & {{Formula}\mspace{14mu} 64} \\{b_{G} = {\begin{bmatrix}b \\b_{T}\end{bmatrix} = \begin{bmatrix}b \\b_{T}\end{bmatrix}}} & {{Formula}\mspace{14mu} 65} \\{{A_{G} \cdot \left\lbrack {x_{p},x_{q},x_{r},\ldots}\mspace{11mu} \right\rbrack^{T}} = {- b_{G}}} & {{Formula}\mspace{14mu} 66}\end{matrix}$

In a gradient-descend like manner the overall problem may be solved bysolving A_(G) for a set of approximated vectors x and using a portion ϵof these values to update the true solution vector x as stated inFormula 32. This true solution vectors x may subsequently be used toupdate initial points m. With those updated points {tilde over (m)}matrix A_(G) may be recomputed. The process continues until congruenceis reached.

In other words, the analyzing unit may be configured to solve a firstand a second optimization criteria for determining the displacementinformation. In other words, the analyzing unit may be configured tosolve a first and a second optimization criteria for the case of havinga 2D camera pattern and for having a 3D camera pattern, wherein thecriteria may differ between the 2D pattern and the 3D pattern.

For the 2D pattern, a possibly non-linear first minimization criteria,may be based on the determination rule

                                     (Formula  15)$\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{{{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}}}}$     with  x = ∑_(l)x_(l)²

as described above. The possibly non-linear second minimization criteriamay be based on the determination rule

${\mspace{650mu} {\left( {{Formula}\mspace{14mu} 36} \right){\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}\sum\limits_{{({p,q,r})} \in \Psi}}}\quad}{{{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{r},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(r)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}}}$

as described above. For the 3D pattern, the possibly non-linear firstminimization criteria may be based on the determination rule

                                     (Formula  15)$\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{{{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}}}}$     with  x = ∑_(l)x_(l)²

and the possibly non-linear second minimization criteria may be based onthe determination rule

$\begin{matrix}{\mspace{79mu} {\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{{g\left( {x_{p},x_{q},x_{r}} \right)}}}}} & \left( {{Formula}\mspace{14mu} 58} \right) \\{\mspace{79mu} {wherein}} & \left( {{Formula}\mspace{14mu} 54} \right) \\{{g\left( {x_{p},x_{q},x_{r}} \right)}:={\left\lbrack {{\overset{\sim}{H}\left( x_{q} \right)} \cdot m_{q}} \right\rbrack_{x} \cdot \left( {{u_{p}^{\#} \cdot T_{{({p,q,r})},1}} + {v_{p}^{\#} \cdot T_{{({p,q,r})},2}} + T_{{({p,q,r})},3}} \right) \cdot \left\lbrack {{\overset{\sim}{H}\left( x_{r} \right)} \cdot m_{r}} \right\rbrack_{x}}} & \; \\{\mspace{79mu} {wherein}} & \left( {{Formula}\mspace{14mu} 55} \right) \\{\mspace{79mu} {m^{\#} = {\begin{bmatrix}u^{\#} & \begin{matrix}v^{\#} & 1\end{matrix}\end{bmatrix}^{T} = {{\overset{\sim}{H}\left( x_{p} \right)} \cdot m}}}} & \;\end{matrix}$

wherein argmin denotes the minimization criteria, and wherein C_(p),C_(g) denote camera positions of cameras p and q, d(C_(p)−C_(g)) is adistance between the camera positions, H is a homography matrix, x_(p),x_(q) and x_(r) are vectors with parameters to be determined so as tominimizing the error for camera p, q and r, m_(q), m_(r) and m_(p,q)refer to a respective minimization result of a previous minimizationiteration, Ω and ψ denote a set of images used for minimization, N is anumber of parameters, v is an image direction, u^(#) and v^(#) denote acalibrated position of an image point along direction u and v obtainedas m_(p) ^(#)=H_(p) m_(p) and T_(i) represents element i of a TrifocalTensor. The explanation of the global optimization problem isrepresented in formula 58, wherein formula 58 may be built from (local)sub-problems given by formula 54.

Alternatively or in addition, the analyzing unit may be configured tosolve linearized minimization criteria, i.e., to use a linearizedversion of the above mentioned criteria. The analyzing unit may thus beconfigured to determine the displacement information minimizing an errorof a linearized first minimization criteria. Accordingly, thedetermining unit may be configured to determine the offset informationminimizing an error of a linearized second minimization criteria. Forthe 2D camera pattern the first linearized minimization criteria may bebased on the determination rule

                                      Formula  67${\left( {{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}} \right)^{2} \approx {{J_{p,q} \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}} + b_{p,q}}} = {{\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}} + {b_{p,q}.}}$

wherein this approximation rule relates to the non-linear minimizationproblem as:

$\begin{matrix}{{{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{{{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}}}}} \approx {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}}}} + b_{p,q}}} = {{J_{B} \cdot \left\lbrack {x_{1},x_{2},x_{3},\ldots \mspace{11mu},x_{n}} \right\rbrack^{T}} + b_{B}}} & {{Formula}\mspace{14mu} 68} \\{\mspace{76mu} {wherein}} & {{Formula}\mspace{14mu} 69} \\{\mspace{76mu} {J_{B} = \begin{bmatrix}J_{pq}^{(p)} & J_{pq}^{(q)} & 0 & 0 & 0 & \ldots & 0 \\J_{pr}^{(p)} & 0 & J_{pr}^{(r)} & 0 & 0 & \ldots & 0 \\0 & J_{qs}^{(q)} & 0 & J_{qs}^{(s)} & 0 & \ldots & 0 \\0 & 0 & J_{rs}^{(r)} & J_{rs}^{(s)} & 0 & \ldots & 0 \\\ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots\end{bmatrix}}} & \; \\{\mspace{76mu} {and}} & {{Formula}\mspace{14mu} 70} \\{\mspace{76mu} {b_{B} = \begin{bmatrix}b_{p,q} & \begin{matrix}b_{p,r} & \begin{matrix}b_{q,s} & \begin{matrix}b_{r,s} & \ldots\end{matrix}\end{matrix}\end{matrix}\end{bmatrix}^{T}}} & \;\end{matrix}$

and for the 2D camera pattern the linearized second minimizationcriteria may be based on the determination rule

                                      Formula  71$\left( {{{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - \left. \quad{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} -}\quad \right.\left. \quad{{\quad\quad}{u\left( {{H\left( {x_{r},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(r)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}} \right)^{2}} \approx}\quad \right.{\quad{{{\left\lbrack {J_{p,q,r}^{(p)}\text{|}J_{p,q,r}^{(q)}\text{|}J_{p,q,r}^{(r)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}} + b_{p,q,r}} = {{J_{p,q,r} \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}} + b_{p,q,r}}}}$

wherein this approximation rule relates to the non-linear minimizationproblem as:

$\begin{matrix}{{{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{{{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{r},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(r)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}}}}} \approx {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{\begin{bmatrix}J_{p,q,r}^{(p)} & \begin{matrix}J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{matrix}\end{bmatrix} \cdot \begin{bmatrix}x_{p}^{T} \\\begin{matrix}x_{q}^{T} \\x_{r}^{T}\end{matrix}\end{bmatrix}}}} + b_{p,q,r}}} = {{J_{T} \cdot \left\lbrack {x_{1},x_{2},x_{3},\ldots \mspace{11mu},x_{n}} \right\rbrack^{T}} + b_{T}}} & {{Formula}\mspace{14mu} 72} \\{\mspace{76mu} {wherein}} & {{Formula}\mspace{14mu} 73} \\{\mspace{76mu} {J_{B} = \begin{bmatrix}J_{pqr}^{(p)} & J_{pqr}^{(q)} & J_{pqr}^{(r)} & 0 & 0 & \ldots & 0 \\J_{prs}^{(p)} & 0 & J_{prs}^{(r)} & J_{prs}^{(s)} & 0 & \ldots & 0 \\0 & J_{qst}^{(q)} & 0 & J_{qst}^{(s)} & J_{qst}^{(t)} & \ldots & 0 \\J_{pst}^{(p)} & 0 & 0 & J_{pst}^{(s)} & J_{pst}^{(t)} & \ldots & 0 \\\ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots\end{bmatrix}}} & \; \\{\mspace{76mu} {and}} & {{Formula}\mspace{14mu} 74} \\{\mspace{76mu} {b_{B} = \begin{bmatrix}b_{p,q,r} & \begin{matrix}b_{p,r,s} & \begin{matrix}b_{q,s,t} & \begin{matrix}b_{p,s,t} & \ldots\end{matrix}\end{matrix}\end{matrix}\end{bmatrix}^{T}}} & \;\end{matrix}$

A solution may be determined, for example as:

$\begin{matrix}{\left\lbrack {x_{1},x_{2},x_{3},\ldots \mspace{11mu},x_{n}} \right\rbrack^{T} = {{- \left( {A_{G}^{T} \cdot A_{G}^{T}} \right)^{- 1}} \cdot A_{G}^{T} \cdot b_{G}}} & {{Formula}\mspace{14mu} 75} \\{wherein} & \; \\{A_{G} = {{\begin{bmatrix}J_{B} \\J_{T}\end{bmatrix}\mspace{14mu} {and}\mspace{14mu} b_{G}} = \begin{bmatrix}b_{B} \\b_{T}\end{bmatrix}}} & \;\end{matrix}$

For the 3D camera pattern the linearized first minimization criteria maybe based on the approximation rule

$\begin{matrix}{{\left( {{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}} \right)^{2} \approx {{J_{p,q} \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}} + b_{p,q}}} = {{\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}} + b_{p,q}}} & {{Formula}\mspace{14mu} 76}\end{matrix}$

wherein this approximation rule relates to the non-linear minimizationproblem as:

$\begin{matrix}{{{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{{{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}}}}} \approx {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}}}} + b_{p,q}}} = {{J_{B} \cdot \left\lbrack {x_{1},x_{2},x_{3},\ldots \mspace{11mu},x_{n}} \right\rbrack^{T}} + b_{B}}} & {{Formula}\mspace{14mu} 77} \\{\mspace{79mu} {wherein}} & {{Formula}\mspace{14mu} 78} \\{\mspace{76mu} {J_{B} = \begin{bmatrix}J_{pq}^{(p)} & J_{pq}^{(q)} & 0 & 0 & 0 & \ldots & 0 \\J_{pr}^{(p)} & 0 & J_{pr}^{(r)} & 0 & 0 & \ldots & 0 \\0 & J_{qs}^{(q)} & 0 & J_{qs}^{(s)} & 0 & \ldots & 0 \\0 & 0 & J_{rs}^{(r)} & J_{rs}^{(s)} & 0 & \ldots & 0 \\\ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots\end{bmatrix}}} & \; \\{\mspace{76mu} {and}} & {{Formula}\mspace{14mu} 79} \\{\mspace{76mu} {b_{B} = \begin{bmatrix}b_{p,q} & \begin{matrix}b_{p,r} & \begin{matrix}b_{q,s} & \begin{matrix}b_{r,s} & \ldots\end{matrix}\end{matrix}\end{matrix}\end{bmatrix}^{T}}} & \;\end{matrix}$

wherein for the 3D camera pattern the linearized second minimizationcriteria is based on the approximation rule

                                      Formula  80${g\left( {x_{p},x_{q},x_{r}} \right)}^{2} = {\left( {\left\lbrack {{\overset{\sim}{H}\left( x_{q} \right)} \cdot m_{q}} \right\rbrack_{x} \cdot \left( {{u_{p}^{\#} \cdot T_{{({p,q,r})},1}} + {v_{p}^{\#} \cdot T_{{({p,q,r})},2}} + T_{{({p,q,r})},3}} \right) \cdot \left\lbrack {{\overset{\sim}{H}\left( x_{r} \right)} \cdot m_{r}} \right\rbrack_{x}} \right)^{2} \approx {{J_{p,q,r} \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}} + b_{p,q,r}}}$

wherein this approximation rule relates to the non-linear minimizationproblem as:

$\begin{matrix}{{{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{{g\left( {x_{p},x_{q},x_{r}} \right)}}}} \approx {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{\begin{bmatrix}J_{p,q,r}^{(p)} & \begin{matrix}J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{matrix}\end{bmatrix} \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}}}} + b_{p,q,r}}} = {{J_{T} \cdot \left\lbrack {x_{1},x_{2},x_{3},\ldots \mspace{11mu},x_{N}} \right\rbrack^{T}} + b_{T}}} & {{Formula}\mspace{14mu} 81} \\{\mspace{79mu} {wherein}} & {{Formula}\mspace{14mu} 82} \\{\mspace{79mu} {J_{T} = \begin{bmatrix}J_{pqr}^{(p)} & J_{pqr}^{(q)} & J_{pqr}^{(r)} & 0 & 0 & \ldots & 0 \\J_{prs}^{(p)} & 0 & J_{prs}^{(r)} & J_{prs}^{(s)} & 0 & \ldots & 0 \\0 & J_{qst}^{(q)} & 0 & J_{qst}^{(s)} & J_{qst}^{(t)} & \ldots & 0 \\J_{pst}^{(p)} & 0 & 0 & J_{pst}^{(s)} & J_{pst}^{(t)} & \ldots & 0 \\\ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots\end{bmatrix}}} & \; \\{\mspace{79mu} {and}} & {{Formula}\mspace{14mu} 83} \\{\mspace{79mu} {b_{T} = \begin{bmatrix}b_{p,q,r}^{T} & b_{p,r,s}^{T} & b_{q,s,t}^{T} & b_{p,s,t}^{T} & \ldots\end{bmatrix}^{T}}} & \;\end{matrix}$

A solution may be determined, for example as:

$\begin{matrix}{\left\lbrack {x_{1},x_{2},x_{3},\ldots \mspace{11mu},x_{N}} \right\rbrack^{T} = {{- \left( {A_{G}^{T} \cdot A_{G}} \right)^{- 1}} \cdot A_{G}^{T} \cdot b_{G}}} & {{Formula}\mspace{14mu} 84} \\{wherein} & {{Formula}\mspace{14mu} 85} \\{A_{G} = {{\begin{bmatrix}A_{B} \\A_{T}\end{bmatrix}\mspace{14mu} {and}\mspace{14mu} b_{G}} = \begin{bmatrix}b_{B} \\b_{T}\end{bmatrix}}} & \;\end{matrix}$

Elements in J_(p,q,r) and b_(p,q,r) are formed from Taylor approximationof g(x_(p),x_(q),x_(r))² according to Formula 58 to Formula 62, andwherein argmin denotes the minimization criteria, and wherein C_(p),C_(p) denote camera positions of cameras p and q, d(C_(p)−C_(q)) is adistance between the camera positions, Ĥ is a homography matrix, {tildeover (x)}_(p), and {tilde over (x)}_(q) are vectors with linearizedparameters to be determined so as to minimizing the error for camera pand q with respect to a constant b_(q,p), J denotes a Jacobian matrix Trepresents a Trifocal Tensor. Subscripts p, q, r, s and t denote indicesreferring to a specific camera.

Thus, the apparatus may be configured to linearly and/or non-linearlyperforming optimization, wherein the selection is up to animplementation. Non-linear methods may allow for using a standardlibrary, wherein the non-linear method may result in a highercomputational effort.

The first optimization problem or criteria may be solved by theanalyzing unit so as to pre-align the images. The second optimizationproblem or criteria may be solved by the determining unit so as toreduce or compensate for remaining differences in disparities.

According to an embodiment, only a coarse setup of the camera system isknown to the algorithm and the exact amount of overlapping image regionswill be approximated or estimated. The analyzing unit is thereforeconfigured to automatically detect corresponding points in all possiblepairs and triplets of images.

A minimum requirement for proper execution according to an example isshown in FIG. 6c . The example refers to an 3×3 example setup of FIG. 4b. Accordingly, the given explanation does not limit the embodiments ofthe present invention.

FIG. 6c shows a schematic representation of a graph that representstriple-matching correspondence points and, by way of example, comprisesnodes N_(i). Each node index i relates to a triplet of cameras C₁,wherein the numbers displayed identify a respective camera C_(i) of thecamera array 40. For example, node N₁ represents a corresponding point 1which is captured by cameras C₁, C₂ and C₃. Each triplet N₁ to N₈ ischaracterized in having a with respect to other triplets possiblyindividual corresponding part of the respective image, i.e., an overlapin the images.

Edges E_(ij) between Nodes N_(i) and N_(j) of the graph and thecorresponding labeling indicate cameras that are incorporated in bothsets of triplet matches. For the sake of simplicity, E_(ij)=E_(ji). Forexample, cameras 1 & 2 are both incorporated in the triplet set {1, 2,3} corresponding to node N₁ and the set {1, 2, 4} corresponding to nodeN₂. Thus, cameras C₁ and C₂ have a (first) overlap containing a (first)correspondence point/object with camera C₃ and have a second overlapcontaining a (second) correspondence point/object with camera C₄.Thereby, a correlation between camera C₃ and camera C₄ may be determinedwith the analyzing unit even if cameras C₃ and C₄ would not have acommon overlap.

It is possible but not needed and even unlikely that a commoncorresponding point is found in each pair of images. However, theembodiments described herein are not limited to a corresponding pointbeing common in each of the cameras or pictures thereof. It issufficient that each camera that is considered in the minimizationproblem is comprised in at least one pair of cameras contained in theset Ω and/or in at least one triplet of cameras contained in ψ. I.e., itmay be sufficient, that for each pair and/or triplet a possiblyindividual corresponding point may be identified.

The analyzing unit may be configured for determining the triplets N,from the pictures taken with the Cameras C_(i) by identifyingcorrespondence points in the respective pictures. The analyzing unit maybe configured for the determining the triplets and/or the graph 60 or arepresentation thereof such that the graph 60 is fully connected.

As shown in FIG. 6c , all cameras (C₁, . . . , C₉) incorporated in thesystem are represented in at least one node and the graph is fullyconnected. In other words there is a path from any node/triplet to anyother node/triplet directly or indirectly by possibly crossing othernodes.

In contrast FIG. 6d shows a similar graph 60′ but with the nodes N₈{5,6,8} and N₉ {5,6,9} being removed such that node N₈ is unconnected tothe rest of the graph. For example, the determination of the analyzingunit is incomplete at the illustrated state.

As can be seen, the node N₈ {6,8,9} is not connected to any other nodethough all cameras (C₁ . . . C₉) are represented in the graph. In thiscase the graph is not fully connected. Hence it is not guaranteed thatthe condition as specified in Formula 36 or Formula 58 is met.

The feature-detection & matching step in the analyzing unit isimplemented to build-up a fully-connected graph or a data representationthereof as shown in FIG. 6c . I.e., it is not needed that the analyzingunit or the determining unit generates a graph or image thereof. It maybe sufficient to generate a data representation thereof, e.g., a matrixor tabular containing the respective information.

A similar condition may also to be met for pairs of cameras. However anyset comprising 3 cameras can be seen as 3 individual pairs of cameras.Therefore any camera that is incorporated in a triplet set is alsoincorporated in at least 2 pairs of correspondence points. Thus, theexplanation given directly relates to 2D and to 3D camera patterns.

The example given in FIG. 6c and FIG. 6d may be referred to as a minimumcase. The apparatus may increase its precision by determining additionalpairs or triplets as long as every camera is represented in the graphand the graph is fully connected. For improving the results, theapparatus may be configured for searching for a higher number ofreference points.

The analyzing unit may determine the reference points so as to becontained in pairs and/or triplets of images.

According to embodiments, the analyzing unit selects or determinesreference/correspondence points in overlapping image regions of thepairs and/or triplets. It is possible that not every possible pair ofimages overlaps. Further, the analyzing unit may be configure forsearching or identifying at least one or up to a specific number ofreference points, e.g., 2, 3, 5, 10 or more. When the specific number isreached, the analyzing unit may be configured for stopping the searchand to thereby determining less than a possible number of referencepoints in the overlap area. This allows saving time and computationaleffort. Further, it may be sufficient to determine a specific referencepoint in the images of cameras of a pair or triplet only, even if thesame reference point is also seen in further images.

When compared to known concepts, it is therefore neither needed to havea known reference object, nor to have a specific object to be containedin every image. It is of advantage to search for correspondingcontent/points in pairs and/or triplets of images as, e.g., at sphericalor cylindrical arrays it becomes increasingly unlikely to have an objectbeing imaged with more than three cameras when increasing the number ofcameras of the array.

FIG. 7 shows a schematic block diagram of an apparatus 70 according toan embodiment that comprises, for example, apparatus 10 and comprisesthe plurality of cameras. The apparatus 70 is configured to apply thecalibration data to the plurality of images 14 so as to obtain aplurality of rectified, i.e., calibrated images 82. For example, theapparatus 70 may comprise a processor 84 for applying the calibrationdata 48. The processor 84 may be configured to combine the calibrationdata 48 with the images 14. The processor 84 may be, for example, theelement 52 of FIG. 1. Thus, not only calibration data may be providedbut also rectified images may be provided. Alternatively or in addition,the apparatus 70 may be configured to determine a depth map of an objectregion such as the object 16. As described in connection with FIG. 1,the object region may comprise a plurality of sub regions, wherein eachcamera 18 a to 18 c of the plurality of cameras may be configured toprovide the corresponding image by projecting one of the plurality ofsub regions and to provide an image based thereon.

FIG. 8 shows a schematic block diagram of a camera system 80 thatcomprises a plurality of cameras 18 a to 18 c and comprises a memory 86having stored thereon the calibration information 48′, being derivedfrom the calibration data 48. For example, the calibration information48′ may be the calibration data 48 or may incorporate at least partthereof. The apparatus 80 may be configured to obtain a plurality ofrectified images 82 and to provide the plurality of rectified images 82,e.g., at an output port of apparatus 80 or to store them in the memory86.

The camera system 80 may further comprise an apparatus 10 that isconfigured to provide for the calibration information 48′.

Some explanations given herein relate to mathematical expressions ofactions implemented by an apparatus and/or a method. It is to be pointedout that such expressions are used for explanatory reasons only so as toclearly define what an apparatus according to the respective embodimentdoes. Therefore, the embodiments do not relate to the mathematicalexpressions but to the actions implemented by the embodiments.

Some embodiments are described in connection with a 2D camera pattern,wherein other embodiments are described in connection with 3D camerapatterns. Further, some embodiments are described in connection withsolving a possibly non-linear optimization, wherein other embodimentsare described in connection with solving a linearized versions thereof.Although being described as separate solutions, further embodiments mayalso provide for combinations. E.g., an analyzing unit may receiveinformation whether the camera pattern is of 2D type or 3D type and mayperform different actions responsive to said information. Further, theanalyzing unit may be blind for said pattern-information and may performboth, 2D determination and 3D determination and may determine a resultof higher quality and may take that result. Further, the analyzing unitmay perform a 3D optimization and may take this result even if thepattern is a 2D pattern because the result is also valid.

Accordingly, the analyzing unit may implement one or both of thenon-linear and linear approach, e.g., by first solving the linearizedformulation and in case of having insufficient results solving thenon-linear formulation.

In the following, additional embodiments and aspects of the inventionwill be described which can be used individually or in combination withany of the features and functionalities and details described herein.

-   1. Apparatus (10; 70) comprising:    -   a first interface (12) for receiving a plurality of partially        overlapping images (14; 14 a-i) of an object (16) from a        corresponding plurality of camera positions (18 a-i) being        arranged along a first and a second direction according to a        camera pattern (30; 40; 40′); an analyzing unit (24) configured        for selecting at least one corresponding reference point (26        ₁-26 ₄) in an overlap area (28) of a set of overlapping images        (14; 14 a-i), and for determining a displacement information        (34) along the first and the second direction (x, y) of the        reference point (26 ₁-26 ₄) in each of the other images of the        set of images, wherein a misalignment of the plurality of images        (14; 14 a-i) along the first and the second direction (x, y) is        compensated by the displacement information (34) so as to obtain        aligned images;    -   a determining unit configured for determining an offset        information (38) between principal points (42 a-c) at the        plurality of camera positions (18 a-i) using at least three        aligned images; and    -   a second interface (46) for providing calibration data (46)        based on the displacement information (34) and based on the        offset information (38), the calibration data allowing for        calibrating the plurality of images so as to comply to the        camera pattern.-   2. The apparatus according to aspect 1, wherein the analyzing unit    (24) is configured for determining the displacement information (34)    using a set of parameters indicating a real condition of the camera    pattern (30; 40; 40′), the parameters comprising a non-linear    relationship with respect to each other, wherein the analyzing unit    (24) is configured to use a linearized version of the set of    parameters and to determine the displacement information (34) by    minimizing an error of the linearized version with respect to a    desired condition of the camera pattern (30; 40; 40′).-   3. The apparatus according to aspect 1 or 2, wherein the analyzing    unit is configured to determine the displacement information    minimizing an error of a first minimization criteria, wherein for a    2D camera pattern the first minimization criteria is based on the    determination rule;

$\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{{{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}}}}$     with  x = ∑_(l)x_(l)²

-   -   wherein the determining unit is configured to determine the        offset information minimizing an error of a second minimization        criteria, wherein for the 2D camera pattern the second        minimization criteria is based on the determination rule

${{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}\sum\limits_{{({p,q})} \in \Psi}}\quad}{{{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{r},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}}}$     with  x = ∑_(l)x_(l)²

-   -   wherein argmin denotes the minimization criteria and p, q and r        denote indices of individual cameras (1, 2, 3, . . . N). C_(p),        C_(q), C_(r) denote camera positions of cameras p, q and r,        d(C_(p)−C_(q)) is a distance between the camera positions,        x_(p), x_(q) and x_(r) are vectors with parameters to be        determined so as to minimizing the error for camera p, q and r,        H(⋅) is a function providing a homography matrix undistorting        points according to a parameter vector x_(p) and camera        positions C_(p) and C_(q), m_(p,q)=[m^((p)) _(p,q) m^((q))        _(p,q)] denotes a set of reference points wherein m^((p)) _(p,q)        corresponds to the respective reference points in camera p,        m_(p,q,r)=[m^((p)) _(p,q,r) m^((q)) _(p,q,r) m^((r)) _(p,q,r)]        denotes a set of reference points wherein m^((p)) _(p,q,r)        corresponds to the respective reference points in camera p,        Functions u(⋅) and v(⋅) extract the horizontal/vertical        component of an image point respectively, the set Ω comprises        all pairs of partially overlapping images; the set ψ comprises        triplets of partially overlapping images;    -   such that each of the N cameras considered in the minimization        problem is comprised in at least one pair of cameras contained        in Ω and at least in one triplet of cameras contained in ψ.

-   4. The apparatus according to one of previous aspects, wherein the    analyzing unit is configured to determine the displacement    information minimizing an error of a linearized first minimization    criteria, wherein for a 2D camera pattern the linearized first    minimization criteria is based on the determination rule;

${\mspace{79mu} {{{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}}}} + b_{p,q}}\mspace{79mu} {{{wherein}\left( {{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}} \right)}^{2} \approx}}\quad}{\quad{{\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}} + b_{p,q}}}$

-   -   wherein the determining unit is configured to determine the        offset information minimizing an error of a linearized second        minimization criteria, wherein for the 2D camera pattern the        linearized second minimization criteria is based on the        determination rule

$\mspace{76mu} {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{\begin{bmatrix}J_{p,q,r}^{(p)} & \begin{matrix}J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{matrix}\end{bmatrix} \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}}}} + b_{p,q,r}}$$\mspace{79mu} {{Wherein}\left( {{{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - \left. \quad{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{r},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}} \right)^{2}} \approx {\quad{{\left\lbrack \begin{matrix}J_{p,q,r}^{(p)} & \begin{matrix}J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{matrix}\end{matrix} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}} + b_{p,q,r}}}} \right.}$

-   -   Wherein argmin denotes the minimization criteria p, q and r        denote indices of individual cameras, C_(p), C_(p), C_(r) denote        camera positions of cameras p, q and r, d(C_(p)−C_(q)) is a        distance between the camera positions, H(⋅) is a function        providing a homography matrix undistorting points according to a        parameter vector x_(p) and camera positions C_(p) and C_(q),        corresponding to camera p with parameter vector x_(p), x_(q) and        x_(r), are vectors with parameters to be determined so as to        minimizing the error for camera p, q and r, m_(p,q)=[m^((p))        _(p,q) m^((q)) _(p,q)] denotes a set of reference points wherein        m^((p)) _(p,q) corresponds to the respective reference points in        camera p, m_(p,q,r)=[m^((p))p,q,r m^((q)) _(p,q,r) m^((r))        _(p,q,r)] denotes a set of reference points wherein m^((p))        _(p,q,r) corresponds to the respective reference points in        camera p, the set Ω comprises all pairs of partially overlapping        images; the set ψ comprises triplets of partially overlapping        images; such that each of the N cameras considered in the        minimization problem is comprised in at least one pair of        cameras contained in Ω and in at least one triplet of cameras        contained in ψ. Functions u(⋅) and v(⋅) extract the        horizontal/vertical component of an image point respectively.        J_(p,q)=[J^((p)) _(p,q) J^((q)) _(p,q)] and J_(p,q,r)=[J^((p))        _(p,q,r) J^((q)) _(p,q,r) J^((r)) _(p,q,r)] denote the Jacobi        matrices for camera pairs and camera triplets, respectively.        J^((p)) _(p,q) and J^((p)) _(p,q,r) denote sub-matrices        corresponding to a camera p. b_(p,q) and b_(p,q,r) denote        residual elements.

-   5. The apparatus according to one of previous aspects, wherein the    analyzing unit is configured to determine the displacement    information minimizing an error of a first minimization criteria,    wherein for a 3D camera pattern the first minimization criteria is    based on the determination rule;

$\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{{{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}}}}$     with  x = ∑_(l)x_(l)²

-   -   wherein the determining unit is configured to determine the        offset information minimizing an error of a second minimization        criteria, wherein for the 3D camera pattern the second        minimization criteria is based on the determination rule

$\mspace{79mu} {\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{{g\left( {x_{p},x_{q},x_{r}} \right)}}}}$     with  x = ∑_(l)x_(l)²      wherein${g\left( {x_{p},x_{q},x_{r}} \right)}:={\left\lbrack {{\overset{\sim}{H}\left( x_{q} \right)} \cdot m_{q}} \right\rbrack_{x} \cdot \left( {{u_{p}^{\#} \cdot T_{{({p,q,r})},1}} + {v_{p}^{\#} \cdot T_{{({p,q,r})},2}} + T_{{({p,q,r})},3}} \right) \cdot \left\lbrack {{\overset{\sim}{H}\left( x_{r} \right)} \cdot m_{r}} \right\rbrack_{x}}$     wherein $\mspace{79mu} {m^{\#} = {\begin{bmatrix}u_{p}^{\#} & v_{p}^{\#} & 1\end{bmatrix}^{T} = {{\overset{\sim}{H}\left( x_{p} \right)} \cdot m}}}$

-   -   wherein argmin denotes the minimization criteria, and p, q and r        denote indices of individual cameras (1, 2, 3, . . . N), C_(p),        C_(p) denote camera positions of cameras p and q, d(C_(p)−C_(q))        is a distance between the camera positions, H(⋅) is a function        providing a homography matrix undistorting points according to a        parameter vector x_(p) and camera positions C_(p) and C_(q),        corresponding to camera p with parameter vector x_(p), x_(q),        x_(q) and x_(r) are vectors with parameters to be determined so        as to minimizing the error for camera p, q and r,        m_(p,q)=[m^((p)) _(p,q) m^((q)) _(p,q)] denotes a set of        reference points wherein m^((p)) _(p,q) corresponds to the        respective reference points in camera p, m_(p,q,r)=[m^((p))p,q,r        m^((q))p,q,r m^((r))p,q,r] denotes a set of reference points        wherein M^((p)) _(p,q,r) corresponds to the respective reference        points in camera p, Function v(⋅) extracts the vertical        component of an image point, the set Ω comprises all pairs of        partially overlapping images; the set ψ comprises triplets of        partially overlapping images; such that each of the N cameras        considered in the minimization problem is comprised at least in        one pair of cameras contained in Ω and in at least one triplet        of cameras contained in ψ. T_((p,q,r),i) represents a matrix        representing the Trifocal Tensor corresponding to camera indices        p, q, and r, m_(p) ^(#) denotes a transformed point according to        a homography matrix H wherein u_(p) ^(#) and v_(p) ^(#) denote        the horizontal and vertical component of a point respectively,        and g models the Trifocal Tensor as defined.

-   6. The apparatus according to one of previous aspects, wherein the    analyzing unit is configured to determine the displacement    information minimizing an error of a linearized first minimization    criteria, wherein for a 3D camera pattern the linearized first    minimization criteria is based on the determination rule;

${\mspace{79mu} {{{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}}}} + b_{p,q}}\mspace{79mu} {{{wherein}\left( {{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}} \right)}^{2} \approx}}\quad}{\quad{{\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}} + b_{p,q}}}$

-   -   wherein the determining unit is configured to determine the        offset information minimizing an error of a linearized second        minimization criteria, wherein for the 3D camera pattern the        linearized second minimization criteria is based on the        determination rule

$\mspace{79mu} {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{\begin{bmatrix}J_{p,q,r}^{(p)} & \begin{matrix}J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{matrix}\end{bmatrix} \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}}}} + b_{p,q,r}}$$\mspace{79mu} {{{wherein}\left( {g\left( {x_{p},x_{q},x_{r}} \right)} \right)}^{2} = {\left( {\left\lbrack {{\overset{\sim}{H}\left( x_{q} \right)} \cdot m_{q}} \right\rbrack_{x} \cdot \left( {{u_{p}^{\#} \cdot T_{{({p,q,r})},1}} + {v_{p}^{\#} \cdot T_{{({p,q,r})},2}} + T_{{({p,q,r})},3}} \right) \cdot \left\lbrack {{\overset{\sim}{H}\left( x_{r} \right)} \cdot m_{r}} \right\rbrack_{x}} \right)^{2} \approx {\quad{{{\begin{bmatrix}J_{p,q,r}^{(p)} & \begin{matrix}J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{matrix}\end{bmatrix} \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}} + {b_{p,q,r}\mspace{79mu} {wherein}\mspace{79mu} m^{\#}}} = {\begin{bmatrix}u_{p}^{\#} & v_{p}^{\#} & 1\end{bmatrix}^{T} = {{\overset{\sim}{H}\left( x_{p} \right)} \cdot m}}}}}}$

-   -   Wherein argmin denotes the minimization criteria p, q and r        denote indices of individual cameras, C_(p), C_(p), C_(r) denote        camera positions of cameras p, q and r, H is a function        providing a homography matrix (compare for Formula 17),        corresponding to camera p with parameter vector x_(p), x_(q),        and x_(r), are vectors with parameters to be determined so as to        minimizing the error for camera p, q and r, m_(p,q)=[m^((p))        _(p,q) m^((q)) _(p,q)] denotes a set of reference points wherein        m^((p)) _(p,q) corresponds to the respective reference points in        camera p, m_(p,q,r)=[m^((p)) _(p,q,r) M^((q)) _(p,q,r) M^((r))        _(p,q,r)] denotes a set of reference points wherein M^((p))        _(p,q,r) corresponds to the respective reference points in        camera p, the set Ω comprises all pairs of partially overlapping        images; the set ψ comprises triplets of partially overlapping        images; such that each of the N cameras considered in the        minimization problem is comprised in at least one pair of        cameras contained in Ω and in at least one triplet of cameras        contained in ψ, Functions u(⋅) and v(⋅) extract the        horizontal/vertical component of an image point respectively,        J_(p,q)=[J^((p)) _(p,q) J^((q)) _(p,q)] and        J_(p,q,r)=[J^((p))p,q,r J^((q))p,q,r J^((r))p,q,r] denote the        Jacobi matrices according to Formulas 22 & 23 and 61 & 62 for        camera pairs and camera triplets, respectively, J^((p)) _(p,q)        and J^((p)) _(p,q,r) denote sub-matrices corresponding to a        camera p, b_(p,q) and b_(p,q,r) denote residual elements,        T_((p,q,r),i) represents a matrix representing the Trifocal        Tensor corresponding to camera indices p, q, and r, m_(p) ^(#)        denotes a transformed point according to a homography H wherein        u_(p) ^(#) and v_(p) ^(#) denote the horizontal and vertical        component of a point respectively.

-   7. The apparatus of one of preceding aspects, wherein the analyzing    unit (24) is configured to determine the displacement information    (34) pairwise for a pair of images of the plurality of images (14;    14 a-i).

-   8. The apparatus of one of preceding aspects, wherein the analyzing    unit (24) is configured to first determine the displacement    information (34) along one of the first and the second direction    (x, y) and to subsequently determine the displacement information    (34) for the other direction.

-   9. The apparatus of one of preceding aspects, wherein the analyzing    unit (24) is configured to determine the displacement information    (34) along a first image direction independent from the displacement    information (34) along a second image direction.

-   10. The apparatus of one of preceding aspects, wherein the    calibration data (48) is based on angles (a, 8, y) that describe a    rotation of the cameras (18 a-i), to a focal length (f) of the    cameras (18 a-i) and to a principal point (42 a-c) of the cameras    (18 a-i), wherein the calibration data (48) does not contain    information indicating a position (C₁-C₉) of cameras of the    plurality of camera positions (18 a-i).

-   11. The apparatus of one of preceding aspects, further comprising at    least one camera being configured to provide the plurality of images    (14; 14 a-i) from the plurality of camera positions (18 a-i),    wherein the apparatus is configured to determine a depth map of an    object region comprising a plurality of subregions (22 a-c), wherein    the at least one camera (18 a-i) is configured to provide the    corresponding image (14 a-i) by projecting one of the plurality of    subregions (22 a-c).

-   12. The apparatus of one of preceding aspects, further comprising at    least one camera being configured to provide the plurality of images    (14; 14 a-i) from the plurality of camera positions (18 a-i),    wherein the apparatus is configured to apply the calibration data    (48) to the plurality of images (14 a-i) so as to obtain a plurality    of calibrated images and to provide the calibrated of rectified    images.

-   13. The apparatus of one of preceding aspects, wherein displacement    information (34) referring to an image of the plurality of images    (14 a-i) comprises at least one of a shift of the image (14 a-i)    along a lateral direction (x, y) and a rotation of the image (14    a-).

-   14. The apparatus of one of preceding aspects, wherein the set of    images is a subset of the plurality of images (14 a-i).

-   15. Camera system (80) comprising at least one camera being    configured to provide a plurality of images (14; 14 a-i) from a    corresponding plurality of camera positions (18 a-i) and comprising    a memory (86) having stored thereon calibration information (48′)    derived from calibration data (48) generated from an apparatus    according to one of previous aspects.

-   16. The camera system of aspect 15, further comprising an apparatus    of one of aspects 1 to 11.

-   17. Method (500; 600) for obtaining calibration data, the method    comprising:    -   receiving (510; 612) a plurality of partially overlapping images        of an object from a corresponding plurality of camera positions        being arranged along a first and a second direction according to        a camera pattern;    -   selecting (520; 616) at least one corresponding reference point        in an overlap area of a set of overlapping images, and        determining a displacement information along the first and the        second direction of the reference point in each of the other        images of the set of images, such that a misalignment of the        plurality of images along the first and the second direction is        compensated by the displacement information so as to obtain        aligned images;    -   determining (530; 606) an offset information between principal        points at the plurality of camera positions using at least three        aligned images; and    -   providing (540; 635) calibration data based on the displacement        information and based on the offset information, the calibration        data allowing for calibrating the plurality of images so as to        comply to the camera pattern.

-   18. Non transitory storage medium having stored thereon a computer    program having a program code for performing, when running on a    computer, a method for obtaining calibration data according to    aspect 17.

Although some aspects have been described in the context of anapparatus, it is clear that these aspects also represent a descriptionof the corresponding method, where a block or device corresponds to amethod step or a feature of a method step. Analogously, aspectsdescribed in the context of a method step also represent a descriptionof a corresponding block or item or feature of a correspondingapparatus.

Depending on certain implementation requirements, embodiments of theinvention can be implemented in hardware or in software. Theimplementation can be performed using a digital storage medium, forexample a floppy disk, a DVD, a CD, a ROM, a PROM, an EPROM, an EEPROMor a FLASH memory, having electronically readable control signals storedthereon, which cooperate (or are capable of cooperating) with aprogrammable computer system such that the respective method isperformed.

Some embodiments according to the invention comprise a data carrierhaving electronically readable control signals, which are capable ofcooperating with a programmable computer system, such that one of themethods described herein is performed.

Generally, embodiments of the present invention can be implemented as acomputer program product with a program code, the program code beingoperative for performing one of the methods when the computer programproduct runs on a computer. The program code may for example be storedon a machine readable carrier.

Other embodiments comprise the computer program for performing one ofthe methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the inventive method is, therefore, acomputer program having a program code for performing one of the methodsdescribed herein, when the computer program runs on a computer.

A further embodiment of the inventive methods is, therefore, a datacarrier (or a digital storage medium, or a computer-readable medium)comprising, recorded thereon, the computer program for performing one ofthe methods described herein.

A further embodiment of the inventive method is, therefore, a datastream or a sequence of signals representing the computer program forperforming one of the methods described herein. The data stream or thesequence of signals may for example be configured to be transferred viaa data communication connection, for example via the Internet.

A further embodiment comprises a processing means, for example acomputer, or a programmable logic device, configured to or adapted toperform one of the methods described herein.

A further embodiment comprises a computer having installed thereon thecomputer program for performing one of the methods described herein.

In some embodiments, a programmable logic device (for example a fieldprogrammable gate array) may be used to perform some or all of thefunctionalities of the methods described herein. In some embodiments, afield programmable gate array may cooperate with a microprocessor inorder to perform one of the methods described herein. Generally, themethods are performed by any hardware apparatus.

While this invention has been described in terms of several advantageousembodiments, there are alterations, permutations, and equivalents whichfall within the scope of this invention. It should also be noted thatthere are many alternative ways of implementing the methods andcompositions of the present invention. It is therefore intended that thefollowing appended claims be interpreted as including all suchalterations, permutations, and equivalents as fall within the truespirit and scope of the present invention.

LITERATURE

-   [1] Zilly F., Riechert C., Müller M., Waizenegger W., Kauff P.,    Determination of rectifying homographies for a camera array,    EP2917895 A1, 2015-   [2] Hartley R., Zissermann A., Multiple View Geometry, Cambridge    University Press, 2003-   [3] Wilburn B., Joshi N., Vaish V., Talvala E., Antunez E., Barth    A., Adams A., Levoy M., Horowitz M., High Performance Imaging Using    Large Camera Arrays, Proc. of ACM SIGGRAPH 2005, Vol. 24, No. 3, pp.    765-776, 2005-   [4] Zhang Z., A flexible new technique for camera calibration,    Pattern Analysis and Machine Intelligence, IEEE Transactions on,    22(11), 1330-1334-   [5] Aerts M., Tytgat D., Macq J., Lievens S., Method and arrangement    for multi-camera calibration, EP 2 375 376 B1, 2013-   [6] Zilly F., Method for the automated analysis, control and    correction of stereoscopic distortions and parameters for 3D-TV    applications, TU Berlin, 2015,    http://dx.doi.org/10.14279/depositonce-4618-   [7] Zilly F., Riechert C., Müller M., Waizenegger W., Sikora T.,    Kauff P., Multi-camera rectification using linearized trifocal    tensor, 21st International Conference on Pattern Recongition (ICPR),    2012-   [8] Kurillo, G., Baker, H., Li, Z., & Bajcsy, R. (2013, June).    Geometric and color calibration of multiview panoramic cameras for    life-size 3D immersive video. In 3D Vision-3DV 2013, 2013    International Conference on (pp. 374-381). IEEE.-   [9] Li, Z., Baker, H., Kurillo, G., & Bajcsy, R. (2010). Projective    epipolar rectification for a linear multi-imager array. In 3DPVT.

1. Apparatus comprising: a first interface for receiving a plurality ofpartially overlapping images of an object from a corresponding pluralityof camera positions being arranged along a first and a second directionaccording to a two-dimensional or three-dimensional camera pattern,wherein patterns comprising camera positions that differ in twodirections are two-dimensional patterns and patterns comprising camerapositions that differ in three directions are three-dimensionalpatterns; an analyzing unit configured for selecting at least onecorresponding reference point in an overlap area of a set of overlappingimages, and for determining a displacement information along the firstand the second direction of the reference point in each of the otherimages of the set of images, wherein a misalignment of the plurality ofimages along the first and the second direction is compensated by thedisplacement information so as to acquire aligned images; a determiningunit configured for determining an offset information between principalpoints at the plurality of camera positions using at least three alignedimages; and a second interface for providing calibration data based onthe displacement information and based on the offset information, thecalibration data allowing for calibrating the plurality of images so asto comply to the camera pattern; wherein the analyzing unit isconfigured to determine the displacement information minimizing an errorof a first minimization criteria, wherein for a 2D camera pattern thefirst minimization criteria is based on the determination rule;$\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{{{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}}}}$     with  x = ∑_(l)x_(l)²  wherein the determining unit isconfigured to determine the offset information minimizing an error of asecond minimization criteria, wherein for the 2D camera pattern thesecond minimization criteria is based on the determination rule${{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Psi}{\quad\quad}}}\quad}{{{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{r},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}}}$     with  x = ∑_(l)x_(l)² wherein argmin denotes the minimizationcriteria and p, q and r denote indices of individual cameras. C_(p),C_(q), C_(r) denote camera positions of cameras p, q and r,d(C_(p)−C_(q)) is a distance between the camera positions, x_(p), x_(q)and x_(r) are vectors with parameters to be determined so as tominimizing the error for camera p, q and r, H(⋅) is a function providinga homography matrix undistorting points according to a parameter vectorx_(p) and camera positions C_(p) and C_(q), m_(p,q)=[m^((p)) _(p,q)m^((q)) _(p,q)] denotes a set of reference points wherein m^((p)) _(p,q)corresponds to the respective reference points in camera p,m_(p,q,r)=[m^((p)) _(p,q,r) m^((q)) _(p,q,r) m^((r)) _(p,q,r)] denotes aset of reference points wherein m^((p)) _(p,q,r) corresponds to therespective reference points in camera p, Functions u(⋅) and v(⋅) extractthe horizontal/vertical component of an image point respectively, theset Ω comprises all pairs of partially overlapping images; the set ψcomprises triplets of partially overlapping images; such that each ofthe N cameras considered in the minimization problem is comprised in atleast one pair of cameras comprised in Ω and at least in one triplet ofcameras comprised in ψ; and/or wherein the analyzing unit is configuredto determine the displacement information minimizing an error of alinearized first minimization criteria, wherein for a 2D camera patternthe linearized first minimization criteria is based on the determinationrule;${\mspace{79mu} {{{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q})} \in \Omega}{\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}}}} + b_{p,q}}\mspace{79mu} {{{wherein}\left( {{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}} \right)}^{2} \approx}}\quad}{\quad{{\left\lbrack {J_{p,q}^{(p)}\text{|}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}} + b_{p,q}}}$ wherein the determining unit is configuredto determine the offset information minimizing an error of a linearizedsecond minimization criteria, wherein for the 2D camera pattern thelinearized second minimization criteria is based on the determinationrule$\mspace{79mu} {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{\arg \; \min}{\sum\limits_{{({p,q,r})} \in \Psi}{\begin{bmatrix}J_{p,q,r}^{(p)} & \begin{matrix}J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{matrix}\end{bmatrix} \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}}}} + b_{p,q,r}}$$\mspace{79mu} {{wherein}\left( {{{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - \left. \quad{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{r},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}} \right)^{2}} \approx {\quad{{\left\lbrack \begin{matrix}J_{p,q,r}^{(p)} & \begin{matrix}J_{p,q,r}^{(q)} & J_{p,q,r}^{(r)}\end{matrix}\end{matrix} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}} + b_{p,q,r}}}} \right.}$ wherein argmin denotes theminimization criteria p, q and r denote indices of individual cameras,C_(p), C_(q), C_(r) denote camera positions of cameras p, q and r,d(C_(p)−C_(q)) is a distance between the camera positions, H(⋅) is afunction providing a homography matrix undistorting points according toa parameter vector x_(p) and camera positions C_(p) and C_(q),corresponding to camera p with parameter vector x_(p), x_(q) and x_(r),are vectors with parameters to be determined so as to minimizing theerror for camera p, q and r, m_(p,q)=[m^((p)) _(p,q) m^((q)) _(p,q)]denotes a set of reference points wherein m^((p)) _(p,q) corresponds tothe respective reference points in camera p, m_(p,q,r)=[m^((p)) _(p,q,r)m^((q)) _(p,q,r) m^((r)) _(p,q,r)] denotes a set of reference pointswherein m^((p)) _(p,q,r) corresponds to the respective reference pointsin camera p, the set Ω comprises all pairs of partially overlappingimages; the set ψ comprises triplets of partially overlapping images;such that each of the N cameras considered in the minimization problemis comprised in at least one pair of cameras comprised in Ω and in atleast one triplet of cameras comprised in ψ, functions u(⋅) and v(⋅)extract the horizontal/vertical component of an image pointrespectively, J_(p,q)=[J^((p)) _(p,q) J^((q)) _(p,q)] andJ_(p,q,r)=[J^((p)) _(p,q,r) J^((q)) _(p,q,r) J^((r)) _(p,q,r)] denotethe Jacobi matrices for camera pairs and camera triplets, respectively,J^((p)) _(p,q) and J^((p)) _(p,q,r) denote sub-matrices corresponding toa camera p, b_(p,q) and b_(p,q,r) denote residual elements.
 2. Theapparatus according to claim 1, wherein the apparatus is configured, fordetermining the displacement information, for using a feature detectionand matching algorithm to detect references on pairs of images and foracquiring matched image coordinates between the images of the pair ofimages.
 3. The apparatus according to claim 1, wherein the apparatus isconfigured for performing a self-calibration without imaging acalibration chart.
 4. The apparatus according to claim 1, wherein theanalyzing unit is configured for determining the displacementinformation using a set of parameters indicating a real condition of thecamera pattern, the parameters comprising a non-linear relationship withrespect to each other, wherein the analyzing unit is configured to use alinearized version of the set of parameters and to determine thedisplacement information by minimizing an error of the linearizedversion with respect to a desired condition of the camera pattern. 5.The apparatus according to claim 1, wherein the analyzing unit isconfigured to determine the displacement information minimizing an errorof a first minimization criteria, wherein for a 3D camera pattern thefirst minimization criteria is based on the determination rule;$\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{argmin}{\sum\limits_{{({p,q})} \in \Omega}{{{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {\quad{{{\quad{v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}}\mspace{14mu} {with}\mspace{14mu} {{x}}} = {\sum_{l}x_{l}^{2}}}}}}}$wherein the determining unit is configured to determine the offsetinformation minimizing an error of a second minimization criteria,wherein for the 3D camera pattern the second minimization criteria isbased on the determination rule$\mspace{79mu} {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{argmin}{\sum\limits_{{({p,q,r})} \in \Psi}{{{g\left( {x_{p},x_{q},x_{r}} \right)}}\mspace{14mu} {with}\mspace{14mu} {x}}}} = {\sum_{l}x_{l}^{2}}}$     wherein${g\left( {x_{p},x_{q},x_{r}} \right)}:={\left\lbrack {{\overset{\sim}{H}\left( x_{q} \right)} \cdot m_{q}} \right\rbrack_{x} \cdot \left( {{u_{p}^{\#} \cdot T_{{({p,q,r})},1}} + {v_{p}^{\#} \cdot T_{{({p,q,r})},2}} + T_{{({p,q,r})},3}} \right) \cdot \left\lbrack {{\overset{\sim}{H}\left( x_{r} \right)} \cdot m_{r}} \right\rbrack_{x}}$     wherein$\mspace{79mu} {{m\#} = {\left\lbrack {u_{p}^{\#}\mspace{20mu} v_{p}^{\#}\mspace{20mu} 1} \right\rbrack^{T} = {{\overset{\sim}{H}\left( x_{p} \right)} \cdot m}}}$wherein argmin denotes the minimization criteria, and p, q and r denoteindices of individual cameras, C_(p), C_(q) denote camera positions ofcameras p and q, d(C_(p)−C_(q)) is a distance between the camerapositions, H(⋅) is a function providing a homography matrix undistortingpoints according to a parameter vector x_(p) and camera positions C_(p)and C_(q), corresponding to camera p with parameter vector x_(p), x_(q),x_(q) and x_(r) are vectors with parameters to be determined so as tominimizing the error for camera p, q and r, m_(p,q)=[m^((p)) _(p,q)m^((q)) _(p,q)] denotes a set of reference points wherein m^((p)) _(p,q)corresponds to the respective reference points in camera p,m_(p,q,r)=[m^((p)) _(p,q,r) m^((q)) _(p,q,r) m^((r)) _(p,q,r)] denotes aset of reference points wherein m^((p)) _(p,q,r) corresponds to therespective reference points in camera p, Function v(⋅) extracts thevertical component of an image point, the set Ω comprises all pairs ofpartially overlapping images; the set ψ comprises triplets of partiallyoverlapping images; such that each of the N cameras considered in theminimization problem is comprised at least in one pair of camerascomprised in Ω and in at least one triplet of cameras comprised in ψ,wherein T_((p,q,r),i) represents a matrix representing the TrifocalTensor corresponding to camera indices p, q, and r, m_(p) ^(#) denotes atransformed point according to a homography matrix H wherein u_(p) ^(#)and v_(p) ^(#) denote the horizontal and vertical component of a pointrespectively, g models the Trifocal Tensor as defined and {tilde over(H)}(x_(i)) is a result of a multiplication of a center matrix K with arotation matrix R_(i) of camera i, correcting a camera's orientationsuch that it corresponds to its ideal orientation as given by an idealcamera setup and the center matrix K modelling a standard camera matrix,K being based on $k = \begin{pmatrix}f_{x} & s & \tau_{x} \\0 & f_{y} & \tau_{y} \\0 & 0 & 1\end{pmatrix}$ with f_(x) and f_(y) denoting focal lengths along a firstand a second direction, s denotes a skew factor and τ_(x) and τ_(y)denote deviations along the first and second direction.
 6. The apparatusaccording to claim 1, wherein the analyzing unit is configured todetermine the displacement information minimizing an error of alinearized first minimization criteria, wherein for a 3D camera patternthe linearized first minimization criteria is based on the determinationrule;$\mspace{79mu} {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{argmin}{\sum\limits_{{({p,q})} \in \Omega}{\left\lbrack {J_{p,q}^{(p)}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}}}} + b_{p,q}}$$\mspace{79mu} {{{wherein}\text{}\left( {{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}} \right)}^{2} \approx {\quad{{\left\lbrack {J_{p,q}^{(p)}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}} + b_{p,q}}}}$ wherein the determining unit is configuredto determine the offset information minimizing an error of a linearizedsecond minimization criteria, wherein for the 3D camera pattern thelinearized second minimization criteria is based on the determinationrule$\mspace{79mu} {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{argmin}{\sum\limits_{{({p,q,r})} \in \Psi}{\left\lbrack {J_{p,q,r}^{(p)}\mspace{20mu} J_{p,q,r}^{(q)}\mspace{14mu} J_{p,q,r}^{(r)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}}}} + b_{p,q,r}}$$\mspace{79mu} {{{wherein}\left( {g\left( {x_{p},x_{q},x_{r}} \right)} \right)}^{2} = \left( {{{\left\lbrack {{\overset{\sim}{H}\left( x_{q} \right)} \cdot m_{q}} \right\rbrack_{x} \cdot \left( {{u_{p}^{\#} \cdot T_{{({p,q,r})},1}} + {{\quad\quad}{v_{p}^{\#} \cdot T_{{({p,q,r})},2}}} + T_{{({p,q,r})},3}} \right) \cdot \left\lbrack {\overset{\sim}{H}\left( {x_{r} \cdot m_{r}} \right\rbrack}_{x} \right)^{2}} \approx {{\left\lbrack {J_{p,q,r}^{(p)}\mspace{20mu} J_{p,q,r}^{(q)}\mspace{20mu} J_{p,q,r}^{(r)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}} + {b_{p,q,r}\mspace{79mu} {wherein}\mspace{79mu} m_{p}^{\#}}}} = {\left\lbrack {u_{p}^{\#}\mspace{20mu} v_{p}^{\#}\mspace{20mu} 1} \right\rbrack^{T} = {{\overset{\sim}{H}\left( x_{p} \right)} \cdot m}}} \right.}$wherein argmin denotes the minimization criteria p, q and r denoteindices of individual cameras, C_(p), C_(p), C_(r) denote camerapositions of cameras p, q and r, H is a function providing a homographymatrixH(x _(p) ,C _(p) ,C _(q))=R ₀ ·K·R _(i) =R ₀ {tilde over (H)}(x _(p))with {tilde over (H)}(x _(p))=K·R _(i) corresponding to camera p withparameter vector x_(p), x_(q), and x_(r), are vectors with parameters tobe determined so as to minimizing the error for camera p, q and r,m_(p,q)=[m^((p)) _(p,q) m^((q)) _(p,q)] denotes a set of referencepoints wherein m^((p)) _(p,q) corresponds to the respective referencepoints in camera p, m_(p,q,r)=[m^((p)) _(p,q,r) m^((q)) _(p,q,r) m^((r))_(p,q,r)] denotes a set of reference points wherein M^((p)) _(p,q,r)corresponds to the respective reference points in camera p, the set Ωcomprises all pairs of partially overlapping images; the set ψ comprisestriplets of partially overlapping images; such that each of the Ncameras considered in the minimization problem is comprised in at leastone pair of cameras comprised in Ω and in at least one triplet ofcameras comprised in ψ, Functions u(⋅) and v(⋅) extract thehorizontal/vertical component of an image point respectively,J_(p,q)=[J^((p)) _(p,q) J^((q)) _(p,q)] and J_(p,q,r)=[J^((p)) _(p,q,r)J^((q)) _(p,q,r) J^((r)) _(p,q,r)] denote the Jacobi matrices accordingtoƒ(x,a):=Tƒ(x,a)=T F _(x,a)(1,0)=F _(x,a)(0)+F _(x,a)′(0)F _(x,a) :

→

:=t>ƒ(a+t·(x−a))and0=[Tg _(1,1)(χ,a),Tg _(1,2)(χ,a), . . . ,Tg _(3,3)(χ,a)]^(T)0=J·χ+b for camera pairs and camera triplets, respectively, J^((p))_(p,q) and J^((p)) _(p,q,r) denote sub-matrices corresponding to acamera p, b_(p,q) and b_(p,q,r) denote residual elements, T_((p,q,r),i)represents a matrix representing the Trifocal Tensor corresponding tocamera indices p, q, and r, m_(p) ^(#) denotes a transformed pointaccording to a homography H, wherein u_(p) ^(#) and v_(p) ^(#) denotethe horizontal and vertical component of a point respectively and {tildeover (H)}(x_(i)) of camera i is a multiplication of a rotation matrixR_(i) correcting a camera's orientation such that it corresponds to itsideal orientation as given by an ideal camera setup and a center matrixK modelling a standard camera matrix based on $K = \begin{pmatrix}f_{x} & s & \tau_{x} \\0 & f_{y} & \tau_{y} \\0 & 0 & 1\end{pmatrix}$ with f_(x) and f_(y) denoting focal lengths along a firstand a second direction, s denotes a skew factor and τ_(x) and τ_(y)denote deviations along the first and second direction; wherein R₀ is anouter rotation matrix, R_(i) is an inner rotation matrix R_(i) of camerai=p, C_(p), C_(q) denote camera positions of cameras p and q and x_(p)relates to a vector, F_(x,a) is a function of t that evaluates thefunction f(x) at the point a+t·(x−a), f(x) is the function toapproximate at a point a, parameters to the function f(x) are comprisedin the parameter vector x, wherein t is a scalar value, a is a vectorwith identical size as x, first order Taylor approximation T of afunction f(x) at the point a is expressed as Taylor expansion of ascalar function F_(x,a)(t), wherein F_(x,a) is expanded at point t₀=0 bycomputing the function value of F_(x,a) with t₀=0 and the value of thefirst order derivative F_(x,a)′=F_(x,a)d/dt also at t₀=0, wherein Tg_(i,j)(χ, a) denotes the Taylor approximation of a function g andindices in g denote an entry in a matrix returned by function g.
 7. Theapparatus of claim 1, wherein the analyzing unit is configured todetermine the displacement information pairwise for a pair of images ofthe plurality of images.
 8. The apparatus of claim 1, wherein theanalyzing unit is configured to first determine the displacementinformation along one of the first and the second direction and tosubsequently determine the displacement information for the otherdirection.
 9. The apparatus of claim 1, wherein the analyzing unit isconfigured to determine the displacement information along a first imagedirection independent from the displacement information along a secondimage direction.
 10. The apparatus of claim 1, wherein the calibrationdata is based on angles that describe a rotation of the cameras, to afocal length of the cameras and to a principal point of the cameras,wherein the calibration data does not comprise information indicating aposition of cameras of the plurality of camera positions.
 11. Theapparatus of claim 1, further comprising at least one camera beingconfigured to provide the plurality of images from the plurality ofcamera positions, wherein the apparatus is configured to determine adepth map of an object region comprising a plurality of subregions,wherein the at least one camera is configured to provide thecorresponding image by projecting one of the plurality of subregions.12. The apparatus of claim 1, further comprising at least one camerabeing configured to provide the plurality of images from the pluralityof camera positions, wherein the apparatus is configured to apply thecalibration data to the plurality of images so as to acquire a pluralityof rectified images and to provide the plurality of rectified images.13. The apparatus of claim 1, wherein displacement information referringto an image of the plurality of images comprises at least one of a shiftof the image along a lateral direction and a rotation of the image. 14.The apparatus of claim 1, wherein the set of images is a subset of theplurality of images.
 15. Camera system comprising at least one camerabeing configured to provide a plurality of images from a correspondingplurality of camera positions and comprising a memory having storedthereon calibration information derived from calibration data generatedfrom an apparatus according to one of previous claims, wherein thecalibration information is the calibration data or incorporates at leastpart thereof.
 16. The camera system of claim 15, further comprising anapparatus of claim
 1. 17. Method for acquiring calibration data, themethod comprising: receiving a plurality of partially overlapping imagesof an object from a corresponding plurality of camera positions beingarranged along a first and a second direction according to atwo-dimensional or three-dimensional camera pattern, wherein patternscomprising camera positions that differ in two directions aretwo-dimensional patterns and patterns comprising camera positions thatdiffer in three directions are three-dimensional patterns; selecting atleast one corresponding reference point in an overlap area of a set ofoverlapping images, and determining a displacement information along thefirst and the second direction of the reference point in each of theother images of the set of images, such that a misalignment of theplurality of images along the first and the second direction iscompensated by the displacement information so as to acquire alignedimages; determining an offset information between principal points atthe plurality of camera positions using at least three aligned images;and providing calibration data based on the displacement information andbased on the offset information, the calibration data allowing forcalibrating the plurality of images so as to comply to the camerapattern; and determining the displacement information minimizing anerror of a first minimization criteria, wherein for a 2D camera patternthe first minimization criteria is based on the determination rule;$\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{argmin}{\sum\limits_{{({p,q})} \in \Omega}{{{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {{\quad{v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}}{\quad\mspace{11mu} {{{with}\mspace{14mu} {{x}}} = {\sum_{l}x_{l}^{2}}}}}}}}$determining the offset information minimizing an error of a secondminimization criteria, wherein for the 2D camera pattern the secondminimization criteria is based on the determination rule$\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{argmin}{\sum\limits_{{({p,q,r})} \in \Psi}{\left( {{{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {\left. \quad{u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {\left. \quad{u\left( {{H\left( {x_{r},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(r)}} \right)} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}} \right.\mspace{14mu} {with}\mspace{14mu} {x}}} = {\sum_{l}x_{l}^{2}}} \right.}}$wherein argmin denotes the minimization criteria and p, q and r denoteindices of individual cameras. C_(p), C_(q), C_(r) denote camerapositions of cameras p, q and r, d(C_(p)−C_(q)) is a distance betweenthe camera positions, x_(p), x_(q) and x_(r) are vectors with parametersto be determined so as to minimizing the error for camera p, q and r,H(⋅) is a function providing a homography matrix undistorting pointsaccording to a parameter vector x_(p) and camera positions C_(p) andC_(q), m_(p,q)=[m^((p)) _(p,q) m^((q)) _(p,q)] denotes a set ofreference points wherein m^((p)) _(p,q) corresponds to the respectivereference points in camera p, m_(p,q,r)=[m^((p)) _(p,q,r) m^((q))_(p,q,r) m^((r)) _(p,q,r)] denotes a set of reference points whereinm^((p)) _(p,q,r) corresponds to the respective reference points incamera p, Functions u(⋅) and v(⋅) extract the horizontal/verticalcomponent of an image point respectively, the set Ω comprises all pairsof partially overlapping images; the set ψ comprises triplets ofpartially overlapping images; such that each of the N cameras consideredin the minimization problem is comprised in at least one pair of camerascomprised in Ω and at least in one triplet of cameras comprised in ψ;and/or determining the displacement information minimizing an error of alinearized first minimization criteria, wherein for a 2D camera patternthe linearized first minimization criteria is based on the determinationrule;$\mspace{79mu} {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{argmin}{\sum\limits_{{({p,q})} \in \Omega}{\left\lbrack {J_{p,q}^{(p)}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}}}} + b_{p,q}}$$\mspace{79mu} {{{wherein}\text{}\left( {{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}} \right)}^{2} \approx {\quad{{\left\lbrack {J_{p,q}^{(p)}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}} + b_{p,q}}}}$ determining the offset informationminimizing an error of a linearized second minimization criteria,wherein for the 2D camera pattern the linearized second minimizationcriteria is based on the determination rule$\mspace{79mu} {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{argmin}{\sum\limits_{{({p,q,r})} \in \Psi}{\left\lbrack {J_{p,q,r}^{(p)}\mspace{20mu} J_{p,q,r}^{(q)}\mspace{20mu} J_{p,q,r}^{(r)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}}}} + b_{p,q,r}}$$\mspace{85mu} {{{wherein}\text{}\left( {{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{r},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(r)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}} \right)}^{2} \approx {\quad{{\left\lbrack {J_{p,q,r}^{(p)}\mspace{20mu} J_{p,q,r}^{(q)}\mspace{20mu} J_{p,q,r}^{(r)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}} + b_{p,q,r}}}}$ wherein argmin denotes the minimizationcriteria p, q and r denote indices of individual cameras, C_(p), C_(q),C_(r) denote camera positions of cameras p, q and r, d(C_(p)−C_(q)) is adistance between the camera positions, H(⋅) is a function providing ahomography matrix undistorting points according to a parameter vectorx_(p) and camera positions C_(p) and C_(q), corresponding to camera pwith parameter vector x_(p), x_(q) and x_(r), are vectors withparameters to be determined so as to minimizing the error for camera p,q and r, m_(p,q)=[m^((p)) _(p,q) m^((q)) _(p,q)] denotes a set ofreference points wherein m^((p)) _(p,q) corresponds to the respectivereference points in camera p, m_(p,q,r)=[m^((p)) _(p,q,r) m^((q))_(p,q,r) m^((r)) _(p,q,r)] denotes a set of reference points whereinm^((p)) _(p,q,r) corresponds to the respective reference points incamera p, the set Ω comprises all pairs of partially overlapping images;the set ψ comprises triplets of partially overlapping images; such thateach of the N cameras considered in the minimization problem iscomprised in at least one pair of cameras comprised in Ω and in at leastone triplet of cameras comprised in ψ, functions u(⋅) and v(⋅) extractthe horizontal/vertical component of an image point respectively,J_(p,q)=[J^((p)) _(p,q) J^((q)) _(p,q)] and J_(p,q,r)=[J^((p)) _(p,q,r)J^((q)) _(p,q,r) J^((r)) _(p,q,r)] denote the Jacobi matrices for camerapairs and camera triplets, respectively, J^((p)) _(p,q) and J^((p))_(p,q,r) denote sub-matrices corresponding to a camera p, b_(p,q) andb_(p,q,r) denote residual elements.
 18. A non-transitory digital storagemedium having a computer program stored thereon to perform the methodfor acquiring calibration data, the method comprising: receiving aplurality of partially overlapping images of an object from acorresponding plurality of camera positions being arranged along a firstand a second direction according to a two-dimensional orthree-dimensional camera pattern, wherein patterns comprising camerapositions that differ in two directions are two-dimensional patterns andpatterns comprising camera positions that differ in three directions arethree-dimensional patterns; selecting at least one correspondingreference point in an overlap area of a set of overlapping images, anddetermining a displacement information along the first and the seconddirection of the reference point in each of the other images of the setof images, such that a misalignment of the plurality of images along thefirst and the second direction is compensated by the displacementinformation so as to acquire aligned images; determining an offsetinformation between principal points at the plurality of camerapositions using at least three aligned images; and providing calibrationdata based on the displacement information and based on the offsetinformation, the calibration data allowing for calibrating the pluralityof images so as to comply to the camera pattern; and determining thedisplacement information minimizing an error of a first minimizationcriteria, wherein for a 2D camera pattern the first minimizationcriteria is based on the determination rule;$\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{argmin}{\sum\limits_{{({p,q})} \in \Omega}{{{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {{\quad{v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}}\; {\quad\mspace{11mu} {{{with}\mspace{14mu} {{x}}} = {\sum_{l}x_{l}^{2}}}}}}}}$determining the offset information minimizing an error of a secondminimization criteria, wherein for the 2D camera pattern the secondminimization criteria is based on the determination rule$\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{argmin}{\sum\limits_{{({p,q,r})} \in \Psi}{\left( {{{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {\left. \quad{u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - {\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {\left. \quad{u\left( {{H\left( {x_{r},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(r)}} \right)} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}}} \right.\mspace{14mu} {with}\mspace{14mu} {x}}} = {\sum_{l}x_{l}^{2}}} \right.}}$wherein argmin denotes the minimization criteria and p, q and r denoteindices of individual cameras. C_(p), C_(q), C_(r) denote camerapositions of cameras p, q and r, d(C_(p)−C_(q)) is a distance betweenthe camera positions, x_(p), x_(q) and x_(r) are vectors with parametersto be determined so as to minimizing the error for camera p, q and r,H(⋅) is a function providing a homography matrix undistorting pointsaccording to a parameter vector x_(p) and camera positions C_(p) andC_(q), m_(p,q)=[m^((p)) _(p,q) m^((q)) _(p,q)] denotes a set ofreference points wherein m^((p)) _(p,q) corresponds to the respectivereference points in camera p, m_(p,q,r)=[m^((p)) _(p,q,r) m^((q))_(p,q,r) m^((r)) _(p,q,r)] denotes a set of reference points whereinm^((p)) _(p,q,r) corresponds to the respective reference points incamera p, Functions u(⋅) and v(⋅) extract the horizontal/verticalcomponent of an image point respectively, the set Ω comprises all pairsof partially overlapping images; the set ψ comprises triplets ofpartially overlapping images; such that each of the N cameras consideredin the minimization problem is comprised in at least one pair of camerascomprised in Ω and at least in one triplet of cameras comprised in ψ;and/or determining the displacement information minimizing an error of alinearized first minimization criteria, wherein for a 2D camera patternthe linearized first minimization criteria is based on the determinationrule;$\mspace{70mu} {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{argmin}{\sum\limits_{{({p,q})} \in \Omega}{\left\lbrack {J_{p,q}^{(p)}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}}}} + b_{p,q}}$      wherein$\left( {{v\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(p)}} \right)} - {v\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q}^{(q)}} \right)}} \right)^{2} \approx {\quad{{\left\lbrack {J_{p,q}^{(p)}J_{p,q}^{(q)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T}\end{bmatrix}} + b_{p,q}}}$ determining the offset informationminimizing an error of a linearized second minimization criteria,wherein for the 2D camera pattern the linearized second minimizationcriteria is based on the determination rule$\mspace{70mu} {{\underset{x_{1},x_{2},x_{3},\ldots,x_{N}}{argmin}{\sum\limits_{{({p,q,r})} \in \Psi}{\left\lbrack {J_{p,q,r}^{(p)}\mspace{20mu} J_{p,q,r}^{(q)}\mspace{20mu} J_{p,q,r}^{(r)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}}}} + b_{p,q,r}}$$\mspace{79mu} {{wherein}\text{}\left( {{{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{q},C_{p},C_{q}} \right)} \cdot m_{p,q,r}^{(q)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{q}} \right)}} - \left. \quad{\left( {{u\left( {{H\left( {x_{p},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(p)}} \right)} - {u\left( {{H\left( {x_{r},C_{p},C_{r}} \right)} \cdot m_{p,q,r}^{(r)}} \right)}} \right)\frac{1}{d\left( {C_{p} - C_{r}} \right)}} \right)^{2}} \approx {\quad{{\left\lbrack {J_{p,q,r}^{(p)}\mspace{20mu} J_{p,q,r}^{(q)}\mspace{20mu} J_{p,q,r}^{(r)}} \right\rbrack \cdot \begin{bmatrix}x_{p}^{T} \\x_{q}^{T} \\x_{r}^{T}\end{bmatrix}} + b_{p,q,r}}}} \right.}$ wherein argmin denotes theminimization criteria p, q and r denote indices of individual cameras,C_(p), C_(q), C_(r) denote camera positions of cameras p, q and r,d(C_(p)−C_(q)) is a distance between the camera positions, H(⋅) is afunction providing a homography matrix undistorting points according toa parameter vector x_(p) and camera positions C_(p) and C_(q),corresponding to camera p with parameter vector x_(p), x_(q) and x_(r),are vectors with parameters to be determined so as to minimizing theerror for camera p, q and r, m_(p,q)=[m^((p)) _(p,q) m^((q)) _(p,q)]denotes a set of reference points wherein m^((p)) _(p,q) corresponds tothe respective reference points in camera p, m_(p,q,r)=[m^((p)) _(p,q,r)m^((q)) _(p,q,r) m^((r)) _(p,q,r)] denotes a set of reference pointswherein m^((p)) _(p,q,r) corresponds to the respective reference pointsin camera p, the set Ω comprises all pairs of partially overlappingimages; the set ψ comprises triplets of partially overlapping images;such that each of the N cameras considered in the minimization problemis comprised in at least one pair of cameras comprised in Ω and in atleast one triplet of cameras comprised in ψ, functions u(⋅) and v(⋅)extract the horizontal/vertical component of an image pointrespectively, J_(p,q)=[J^((p)) _(p,q) J^((q)) _(p,q)] andJ_(p,q,r)=[J^((p)) _(p,q,r) J^((q)) _(p,q,r) J^((r)) _(p,q,r)] denotethe Jacobi matrices for camera pairs and camera triplets, respectively,J^((p)) _(p,q) and J^((p)) _(p,q,r) denote sub-matrices corresponding toa camera p, b_(p,q) and b_(p,q), denote residual elements, when saidcomputer program is run by a computer.